Leopold - thanks for a clear, vivid, candid, and galavanizing post. I agree with about 80% of it. 

However, I don't agree with your central premise that alignment is solvable. We want it to be solvable. We believe that we need it to be solvable (or else, God forbid, we might have to actually stop AI development for a few decades or centuries). 

But that doesn't mean it is solvable. And we have, in my opinion, some pretty compelling reasons to think that it not solvable even in principle, (1) given the diversity, complexity, and ideological nature of many human values (which I've written about in other EA Forum posts, and elsewhere), (2) given the deep game-theoretic conflicts between human individuals, groups, companies, and nation-states (which cannot be waved away by invoking Coherent Extrapolated Volition, or 'dontkilleveryoneism', or any other notion that sweeps people's profoundly divergent interests under the carpet), and (3) given that humans are not the only sentient stakeholder species that AI would need to be aligned with (advanced AI will have implications for every other of the 65,000 vertebrate species on Earth, and most of the 1,000,000+ invertebrate species, one way or another). 

Human individuals aren't aligned with each other. Companies aren't aligned with each other. Nation-states aren't aligned with each other. Other animal species aren't aligned with humans, or with each other. There is no reason to expect that any AI systems could be 'aligned' with the totality of other sentient life on Earth. Our Bayesian prior, based on the simple fact that different sentient beings have different interests, values, goals, and preferences, must be that AI alignment with 'humanity in general', or 'sentient life in general', is simply not possible. Sad, but true.

I worry that 'AI alignment' as a concept, or narrative, or aspiration, is just promising enough that it encourages the AI industry to charge full steam ahead (in hopes that alignment will be 'solved' before AI advances to much more dangerous capabilities), but it is not delivering nearly enough workable solutions to make their reckless accelerationism safe. We are getting the worst of both worlds -- a credible illusion of a path towards safety, without any actual increase in safety.

In other words, the assumption that 'alignment is solvable' might be a very dangerous X-risk amplifier, in its own right. It emboldens the AI industry to accelerate. It gives EAs (probably) false hope that some clever technical solution can make humans all aligned with each other, and make machine intelligences aligned with organic intelligences. It gives ordinary citizens, politicians, regulators, and journalists the impression that some very smart people are working very hard on making AI safe, in ways that will probably work. It may be leading China to assume that some clever Americans are already handling all those thorny X-risk issues, such that China doesn't really need to duplicate those ongoing AI safety efforts, and will be able to just copy our alignment solutions once we get them.

If we take seriously the possibility that alignment might not be solvable, we need to rethink our whole EA strategy for reducing AI X-risk. This might entail EAs putting a much stronger emphasis on slowing or stopping further AI development, at least for a while. We are continually told that 'AI is inevitable', 'the genie is out of the bottle', 'regulation won't work', etc. I think too many of us buy into the over-pessimistic view that there's absolutely nothing we can do to stop AI development, while also buying into the over-optimistic view that alignment is possible -- if we just recruit more talent, work a little more, get a few more grants, think really hard, etc. 

I think we should reverse these optimisms and pessimisms. We need to rediscover some optimism that the 8 billion people on Earth can pause, slow, handicap, or stop AI development by the 100,000 or so AI researchers, devs, and entrepreneurs that are driving us straight into a Great Filter. But we need to rediscover some pessimism about the concept of 'AI alignment' itself. 

In my view, the burden of proof should be on those who think that 'AI alignment with human values in general' is a solvable problem. I have seen no coherent argument that it is solvable. I've just seen people desperate to believe that it is solvable. But that's mostly because the alternative seems so alarming, i.e., the idea that (1) the AI industry is increasingly imposing existential risks on us all, (2) it has a lot of money, power, talent, influence, and hubris, (3) it will not slow down unless we make it slow down, and (4) slowing it down will require EAs to shift to a whole different set of strategies, tactics, priorities, and mind-sets than we had been developing within the 'alignment' paradigm. 

This is some of the finest writing I've seen on AI alignment which both (a) covers technical content , and (b) is accessible to a non-technical audience. 

I particularly liked the fact that the content was opinionated; I think it's easier to engage with content when the author takes a stance rather than just hedges their bets throughout.

This comes late, but I appreciate this post and am curating it. I think the core message is an important one, some sections can help people develop intuitions for what the problems are, and the post is written in an accessible way (which is often not the case for AI safety-related posts). As others noted, the post also made a bunch of specific claims that others can disagree with as opposed to saying vague things or hedging a lot, which I also appreciate (see also epistemic legibility). 

I share Charlie Guthmann's question here: I get the sense that some work is in a fuzzy grey area between alignment and capabilities, so comparing the amount of work being done on safety vs. capabilities is difficult. I should also note that I don't think all capabilities work can be defended as safety-relevant (see also my own post on safety-washing). 

...

Quick note: I know Leopold — I don't think this influenced my decision to curate the post, but FYI. 

This was enlightening and convincing. Plus a great read!

I'm late to this, but I'm surprised that this post doesn't acknowledge the approach of inverse reinforcement learning (IRL) which Stuart Russell discussed on the 80,000 Hours podcast and which also featured in his book Human Compatible. 

I'm no AI expert, but this approach seems to me like it avoids the "as these models become superhuman, humans won’t be able to reliably supervise their outputs" problem, as a superhuman AI using IRL doesn't have to be supervised, it just observes us and through doing so better understands our values. 

I'm generally surprised at the lack of discussion of IRL in the community. When one of the world leaders in AI says a particular approach in AI alignment is our best hope, shouldn't we listen to them?

I appreciate that you are putting out numbers and explain the current research landscape, but I am missing clear actions.

The closest you are coming to proposing them is here:

We need a concerted effort that matches the gravity of the challenge. The best ML researchers in the world should be working on this! There should be billion-dollar, large-scale efforts with the scale and ambition of Operation Warp Speed or the moon landing or even OpenAI’s GPT-4 team itself working on this problem.[17] Right now, there’s too much fretting, too much idle talk, and way too little “let’s roll up our sleeves and actually solve this problem.”

But that still isn't an action plan. Say you convince me, most of the EA Forum and half of all university educated professionals in your city that this is a big deal. What, concretely, should we do now?

It seems plausible that there are ≥100,000 researchers working on ML/AI in total. That’s a ratio of ~300:1, capabilities researchers:AGI safety researchers.

Barely anyone is going for the throat of solving the core difficulties of scalable alignment. Many of the people who are working on alignment are doing blue-sky theory, pretty disconnected from actual ML models.

One question I'm always left with is: what is the boundary between being an AGI safety researcher and a capabilities researcher?

For instance, My friend is getting his PhD in machine learning, he barely knows about EA or LW, and definitely wouldn't call himself a safety researcher. However, when I talk to him, it seems like the vast majority of his work deals with figuring out how ML systems act when put in foreign situations wrt the training data. 

I can't claim to really understand what he is doing but it sounds to me a lot like safety research. And it's not clear to me this is some "blue-sky theory". A lot of the work he does is high-level maths proofs, but he also does lots of interfacing with ml systems and testing stuff on them. Is it fair to call my friend a capabilities researcher?

(Importantly, from my understanding, this isn’t OpenAI being evil or anything like that—OpenAI would love to hire more alignment researchers, but there just aren’t many great researchers out there focusing on this problem.)

Thank you emphasizing you're not implying OpenAI is evil only because some practices at OpenAI may be inadequate. I feel like I shouldn't have to thank you for that, though I do, just to emphasize how backwards the thinking and discourse in the AI safety/alignment community often is when a pall of fear and paranoia is cast on all AI capabilities researchers. 

During a recent conversation about AI alignment with a few others, when I expressed a casual opinion about how AGI labs have some particularly mistaken practice, I too felt a need to clarify I didn't mean to imply that Sundar Pichai or Sam Altman are evil because of it. 

I don't even remember right now what that point of criticism I made was. I don't remember if that conversation was last week or the week before. It hasn't stuck in my mind because it didn't feel that important. It was an offhand comment about a relatively minor mistake AGI labs are making, tangential to the main argument I was trying to make. 

Yet it's telling that I felt a need to clarify I wasn't implying OpenAI or DeepMind is evil, during even just a private conversation. It's telling that you've felt a need to do that in this post. It's a sign of a serious problem in the mindset of at least a minority of the AI safety/alignment community. 

Another outstanding feature of this post is how you've mustered the effort to explain at all why you consider different approaches to alignment to be inadequate. This distinguishes your post from others like it from the the last year. Others who've tried to get across the same point you're making have, instead of explaining their disagreements, have generally alleged almost everyone else in entire field of AI alignment are literally insane. 

That's not helpful for a few reasons. Such a claim is probably not true. It'd be harder to make a more intellectually lazy or unconvincing argument. It counts as someone making a bold, senseless attempt to, arguably, dehumanize hundreds of their peers. 

This isn't just a negligible error from somebody recognized as part of a hyperbolic fringe in AI safety/alignment community. It's direly counterproductive when it comes from leading rationalists, like Eliezer Yudkowsky and Oliver Habryka, who wield great influence in their own right, and are taken very seriously by hundreds of other people. Thank you for writing this post as corrective to that kind of mistake a lot of your close allies have been making too.

I have so far gotten the same impression that making RLHF work as a strategy by iteratively and kind of gradually scaling it in a very operationally secure way seems like maybe the most promising approach. My viewpoint right now still remains as the one you've expressed about how, while as much as the RLHF++ has going for it in a relative sense, in leaves a lot to be desired in an absolute sense in light of the alignment/control problem for AGI. 

Overall, I really appreciate how this post condenses well in detail what is increasingly common knowledge about just how inadequate are the sum total of major approaches being taken to alignment. I've read analyses with the same current conclusion from several other AGI safety/alignment researchers during the last year or two. Yet where I hit a wall is my strong sense that any alternative approaches could just as easily succumb to most if not all of the same major pitfalls you list the RLHF++ approach of having to contend with. In that sense, I also feel most of your points are redundant, To get specific about how your criticisms of RLHF apply to all the other alignment approaches as well... 

This currently feels way too much like “improvise as we go along and cross our fingers” to be Plan A; this should be Plan B or Plan E.

Whether it's an approach inspired by the paradigms established in light of Christiano, Yudkowsky, interpretability research, or elsewise, I've gotten the sense essentially all alignment researchers honestly feel the same way about whatever approach to RLHF they're taking.

"It might well not work. I expect this to harvest a bunch of low-hanging fruit[...]This really shouldn't be our only plan.

I understand how it feels like, based on how some people tend to talk about RLHF, and sometimes interpretability, they're implying or suggesting that we'll be fine with just this one approach. At the same time, as far as I'm aware, when you get behind any hype, almost everyone admits that whatever particular approach to alignment they're taking may fail to generalize and shouldn't be the only plan. 

It rests on pretty unclear empirical assumptions on how crunchtime will go.

I've gotten the sense that the empirical assumption for how crunchtime will go among researchers taking the RLHF approach is, for lack of a better term, kind of a medium-term forecast for the date of the tipping point for AGI, i.e., probably at least between 2030 and 2040, as opposed to between 2025 and 2030. 

Given this or that certain chain/sequence of logical assumptions about the trajectory or acceleration of capabilities research, there of course is an intuitive case to be made, on rational-theoretic grounds, for acting/operating under the presumption in practice that forecasts of short(er) AGI timelines, e.g., between 1 and 5 years out, are just correct and the most accurate. 

At the same time, such models for timeline and/or trajectory towards AGI anyone could, just as easily, be totally wrong. Those research teams most dedicated to really solving the control problem for transformative/general AI with the shortest timelines are also acting under assumptions derived from models that also severely lacking any empirical basis. 

As far as I'm aware, there is a combined set of several for-profit startups, and non-profit research organizations, that have been trialing state-of-the-art approaches for prediction markets and forecasting methodologies, especially timelines and trajectories of capabilities research for transformative/general AI. 

During the last few years, they've altogether received, at least, a few million dollars to run so many experiments to determine how to achieve more empirically based models for AI timelines or trajectories. While there may potentially be valuable insights for empirical forecasting methods overall, I'm not aware of any results at all vindicating, literally, any theoretical model for capabilities forecasting. 

I’m not sure this plan puts us on track to get to a place where we can be confident that scalable alignment is solved. By default, I’d guess we’d end up in a fairly ambiguous situation.

This is yet just another criticism of the RLHF approach that I understand as just as easily applying to any approach to alignment you've mentioned, and even every remotely significant approach to alignment you didn't mention but I've also encountered.

You also mentioned, for both the relatively cohesive (set of) approach(es) inspired by Christiano's research, or for more idiosyncratic approaches, a la MIRI, you perceive to be a dead end the very abstract, and almost purely mathematical, approach being taken. That's an understandable and sympathetic take. All things being equal, I'd agree with your proposal for what should be done instead:

We need a concerted effort that matches the gravity of the challenge. The best ML researchers in the world should be working on this! There should be billion-dollar, large-scale efforts with the scale and ambition of Operation Warp Speed or the moon landing or even OpenAI’s GPT-4 team itself working on this problem.

Unfortunately, all things are not equal. The societies we live in will. for foreseeable future. keep operating on a set of very unfortunate incentive structures. 

To so strongly invest into ML-based approaches to alignment research, in practice, often entails working in some capacity of advancing capabilities research even more, especially in industry and the private sector. That's a major reason why, regardless of whatever ways they might be superior, ML-based approaches to alignment are often eschewed. 

I.e., most conscientious alignment researchers don't feel like their field is ready to pivot so fully to ML-based approaches to alignment, without in the process increasing whatever existential risk super-human AGI might pose to humanity, as opposed to decreasing such risk. As harsh as I'm maybe being, I also think the most novel and valuable propositions in this post are your own you've downplayed:

For example, I’m really excited about work like this recent paper (paper, blog post on broader vision), which prototypes a method to detect “whether a model is being honest” via unsupervised methods. More than just this specific result, I’m excited about the style:

• Use conceptual thinking to identify methods that might plausibly scale to superhuman methods (here: unsupervised methods, which don’t rely on human supervision)
• Empirically test this with current models.

I think there’s a lot more to do in this vein—carefully thinking about empirical setups that are analogous to the core difficulties of scalable alignment, and then empirically testing and iterating on relevant ML methods.

My one recommendation is that you don't dwell any longer on so many things in AI alignment as a field most alignment researchers already acknowledge, and get down your proposals for taking an evidence-based approach to expanding the robustness of alignment of unsupervised systems. That's as exciting a new research direction I've heard of in the last year too!

Copy-pasting here from LW.

Sorry but my rough impression from the post is you seem to be at least as confused about where the difficulties are as average of alignment researchers you think are not on the ball - and the style of somewhat strawmanning everyone & strong words is a bit irritating.

Maybe I'm getting it wrong, but it seems the model you have for why everyone is not on the ball is something like "people are approaching it too much from a theory perspective, and promising approach is very close to how empirical ML capabilities research works" & "this is a type of problem where you can just throw money at it and attract better ML talent".

I don't think these two insights are promising.

Also, again, maybe I'm getting it wrong, but I'm confused how similar you are imagining the current systems to be to the dangerous systems. It seems either the superhuman-level problems (eg not lying in a way no human can recognize) are somewhat continuous with current problems (eg not lying), and in that case it is possible to study them empirically. Or they are not.  But different parts of the post seem to point in different directions. (Personally I think the problem is somewhat continuous, but many of the human-in-the-loop solutions are not, and just break down.)

Also, with what you find promising I'm confused what do you think the 'real science'  to aim for is  - on one hand it seems you think the closer the thing is to how ML is done in practice the more real science it is. On the other hand, in your view all deep learning progress has been empirical, often via dumb hacks and intuitions (this isn't true imo). 

I can't get past the feeling that all the doom will flow through an asymptote of imperfect alignment. How can scalable alignment ever be watertight enough for x-risk to drop to insignificant levels? Especially given the (ML-based) engineering approach suggested. It sounds like formal, verifiable, proofs of existential safety won't ever be an end product of all this. How long do we last in a world like that, where AI capabilities continue improving up to physical limits? Can the acute risk period really be brought to a close this way?

TL;DR: I totally agree with the general spirit of this post, we need people to solve alignment, and we're not on track. Go and work on alignment but before you do, try to engage with the existing research, there are reasons why it exists. There are a lot of things not getting worked on within AI alignment research, and I can almost guarantee you that within six months to a year, you can find things that people haven't worked on. 

So go and find these underexplored areas in a way where you engage with what people have done before you!

There’s no secret elite SEAL team coming to save the day. This is it. We’re not on track.

If timelines are short and we don’t get our act together, we’re in a lot of trouble. Scalable alignment—aligning superhuman AGI systems—is a real, unsolved problem. It’s quite simple: current alignment techniques rely on human supervision, but as models become superhuman, humans won’t be able to reliably supervise them.

But my pessimism on the current state of alignment research very much doesn’t mean I’m an Eliezer-style doomer. Quite the opposite, I’m optimistic. I think scalable alignment is a solvable problem—and it’s an ML problem, one we can do real science on as our models get more advanced. But we gotta stop fucking around. We need an effort that matches the gravity of the challenge.^[1]

I also agree in that Eliezer's style of doom seems uncalled for and that this is a solvable but difficult problem. My personal p(doom) is something around 20%, and I think this seems quite reasonable. 

Barely anyone is going for the throat of solving the core difficulties of scalable alignment. Many of the people who are working on alignment are doing blue-sky theory, pretty disconnected from actual ML models. Most of the rest are doing work that’s vaguely related, hoping it will somehow be useful, or working on techniques that might work now but predictably fail to work for superhuman systems.

Now I do want to give pushback on this claim as I see a lot of people who haven't fully engaged with the more theoretical alignment landscape making this claim. There are only 300 people working on alignment, but those people are actually doing things, and most of them aren't doing blue in the sky theory.

A note on the ARC claim:

But his research now (“heuristic arguments”) is roughly “trying to solve alignment via galaxy-brained math proofs.” As much as I respect and appreciate Paul, I’m really skeptical of this: basically all deep learning progress has been empirical, often via dumb hacks^[3] and intuitions, rather than sophisticated theory. My baseline expectation is that aligning deep learning systems will be achieved similarly.^[4] 

This is essentially a claim about the methodology of science in that working on existing systems gives more information and breakthroughs compared to working on a blue-sky theory. The current hypothesis for this is that it is just a lot more information-rich to do real-world research. This is, however, not the only way to get real-world feedback loops. Christiano is not working on blue sky theory; he's using real-world feedback loops in a different way; he looks at the real world and looks for information that's already there. 

A discovery of this type is, for example, the tragedy of the commons; whilst we could have created computer simulations to see the process in action, it's 10x easier to look at the world and see the real-time failures. He tells stories and sees where they fail in the future as his research methodology. This gives bits of information on where to do future experiments, like how we would be able to tell that humans would fail to stop overfishing without actually running an experiment on it.

This is also what John Wentworth does with his research; he looks at the real world as a reference frame which is quite rich in information. Now a good question is why we haven't seen that many empirical predictions from Agent Foundations. I believe it is because alignment is quite hard, and specifically, it is hard to define agency in a satisfactory way due to some really fuzzy problems (boundaries, among others) and, therefore, hard to make predictions. 

We don't want to mathematize things too early either, as doing so would put us into a predefined reference frame that it might be hard to escape from. We want to find the right ballpark for agents since if we fail we might base evaluations on something that turns out to be false. 

In general, there's a difference in the types of problems in alignment and empirical ML; the reference class of a "sharp-left turn" is different from something empirically verifiable as it is unclearly defined, so a good question is how we should turn one into the other. This question of how we take recursive self-improvement, inner misalignment and agent foundations into empirically verifiable ML experiments is actually something that most of the people I know in AI Alignment are currently actively working on.

This post from Alexander Turner is a great example of doing this as they try "just retargeting the search"

Other people are trying other things, such as bounding the maximisation in RL into quantilisers. This would, in turn, make AI more "content" with not maximising. (fun parallel to how utilitarianism shouldn't be unbounded)

I could go on with examples, but what I really want to say here is that alignment researchers are doing things; it's just hard to realise why they're doing things when you're not doing alignment research yourself. (If you want to start, book my calendly and I might be able to help you.)

So what does this mean for an average person? You can make a huge difference by going in and engaging with arguments and coming up with counter-examples, experiments and theories of what is actually going on. 

I just want to say that it's most likely paramount to engage with the existing alignment research landscape before as it's free information and easy to fall into traps if you don't. (a good resource for avoiding some traps is John's Why Not Just sequence) 

There's a couple of years worth of research there; it is not worth rediscovering from the ground up. Still, this shouldn't stop you, go and do it; you don't need a hero licence.

I'm very supportive of this post. Also I will shamelessly share here a sequence I posted in February called "The Engineer's Interpretability Sequence". One of the main messages of the sequence could be described as how existing mechanistic interpretability research is not on the ball. 

https://www.alignmentforum.org/s/a6ne2ve5uturEEQK7 

I agree with this post. I've been reading many more papers since first entering this field because I've been increasingly convinced of the value of treating alignment as an engineering problem and pulling insights from the literature. I've also been trying to do more thinking about how to update on the current paradigm from the classic Yud and Bostrom alignment arguments. In this respect, I applaud Quintin Pope for his work.

This week, I will send a grant proposal to continue my work in alignment. I'd be grateful if you could look at my proposal and provide some critique. It would be great to have an outside view (like yours) to give feedback on it.

Current short summary: "This project comprises two main interrelated components: accelerating AI alignment research by integrating large language models (LLMs) into a research system, and conducting direct work on alignment with a focus on interpretability and steering the training process towards aligned AI. The "accelerating alignment" agenda aims to impact both conceptual and empirical aspects of alignment research, with the ambitious long-term goal of providing a massive speed-up and unlocking breakthroughs in the field. The project also includes work in interpretability (using LLMs for interpreting models; auto-interpretability), understanding agency in the current deep learning paradigm, and designing a robustly aligned training process. The tools built will integrate seamlessly into the larger alignment ecosystem. The project serves as a testing ground for potentially building an organization focused on using language models to accelerate alignment work."

Please send me a DM if you'd like to give feedback!

[epistemic status: half-joking]

There’s no secret elite SEAL team coming to save the day.

Are there any organized groups of alignment researchers who serve as a not-so-secret, normal civilian equivalent of a SEAL team trying their best to save the day, while also trying to make no promises of being some kind of elite, hyper-competent super-team?

The challenge isn’t figuring out some complicated, nuanced utility function that “represents human values”; the challenge is getting AIs to do what it says on the tin—to reliably do whatever a human operator tells them to do.

IMO, this implies we need to design AI systems so that they satisfice rather than maximize: perform a requested task at a requested performance level but no better than that and with a requested probability but no more likely than that.

Great post. What I find most surprising is how small the scalable alignment team at OpenAI is. Though similar teams in DeepMind and Anthropic are probably bigger.

The challenge isn’t figuring out some complicated, nuanced utility function that “represents human values”; the challenge is getting AIs to do what it says on the tin—to reliably do whatever a human operator tells them to do.

Why do you think this? I infer for what I've seen written in other posts and comments that this is a common belief but I don't find the reasons why. 

The fact that there are specific really difficult problems with aligning ML systems doesn't mean that the original really difficult problem with finding and specifying the objectives that we want for a superintelligence were solved.

I hate it because it makes it seems like alignment is a technical problem that can be solved by a single team and as you put it in your other post we should just race and win against the bad guys. 

I could try to envision what type of AI you are thinking of and how would you use it, but I would prefer if you tell me. So, what would you ask your aligned AGI to do and how would it interpret that? And how are you so sure that most alignment researchers would ask it the same things as you?

(On phone so rushed reply)

Thanks for this, it's well written and compelling.

Who needs to do what differently do you think?

Do you have object level recommendations for specific audiences?

For instance a call for more of a specific type of projects to increase the number of people working on AI Safety, their quality of work or coordination etc?

I'm no expert on this at all but this is very interesting. I wonder if the key to alignment is some kind of inductive alignment where rather than designing a superintelligent system T_N in a vacuum you design a series of increasingly intelligent systems T_0..T_N where the alignment is inbuilt at each level and a human aligns the basic T_0. i.e. you build T_1 as a small advancement of T_0 such that it can be aligned by T_0 and so on.

Loved the language in the post! To the point without having to use unnecessary jargon.

There are two things I'd like you to elaborate on if possible:

> "the challenge is getting AIs to do what it says on the tin—to reliably do whatever a human operator tells them to do."

If I understand correctly you imply that there is still a human operator to a superhuman AGI, do you think this is the way that alignment will work out? What I see is that humans have flaws, do we really want to give a "genie" / extremely powerful tool to humans that even already struggle with the powerful tools that they have? At least right now these powerful tools are in the hands of the more responsible few, but if it becomes more widely accessible that's very different.

What do you think of going the direction of developing a "Guardian AI", which would still solve the alignment problem using the tools of ML, but involving humans giving up control of the alignment?

The second one is more practical, which action do you think one should take. I've of course read the recommendations that other people have put out there so far, but would be curious to hear your take on this.