The Problem of Knowledge and Ignorance in Economics
RCTs, Rationality, Shackle and Critical Realist Economics
Studying economics and studying methodology are inseparable. Most geniuses who pondered on the questions of economics, also paused to ask what method was apt in studying behavior. Sample this diversity for example (1, 2, 3, 4, 5, 6, 7). The question has three threads. How much can we know about the world? How to understand the world? How to use knowledge to explain the world? To start writing on this blog, in this article I share some edited notes on how I think about methodology in Economics. To begin with I will try to review struggles with context, followed by time and uncertainty, the role of modeling and empirics, institutions and culture and then make a case for critical realist economics. I will show how our empirical methods are limited. How they explore the mechanisms playing out in the actual world in a limited sense. I will then show that the actual world is further distinct from the real world. A difference we can only incompletely perceive as a black box of many factors. I will keep rewriting this and add and remove parts as my thinking evolves.
Contents
II. Time, thoughts and the ghost of Shackle
III. Probability and Living by the bell curve
IV. Crossing the river by feeling the stones
I – Rationality and context
Economics: A primer
Ashish Kulkarni, our professor, following Alex and Tyler, professes that economics is the art of making the most of life. I quite like that definition. But what does that mean for an academic or a social thinker? We can anticipate the rational (technically defined here) behavior of an individual acting optimally in terms of choosing what is more valuable over what is less valuable, recognizing opportunity costs, time horizons, incentives, and predictable consequences in choosing. But matters indeed get more complex in the aggregate.
What’s rational for a person might not be for others depending on our subjective or incomplete assessment of their circumstances. The question of methodology is then born out of angst. How do we theorize the general? To paraphrase Friedman, ‘A photograph of the world is hardly a theory about it’. Selective specialization of ideas into concepts and relations becomes necessary, assuming away is critical for a theory to emerge. The purpose of theorisation could be any, the method is critical.
We will take two extreme methods in Economics to understand what this might look like in practice. Model-testing and RCTs. I choose these two because their usage is common and they have a legacy of usage now. On the spectrum between these two you could include a variety of other methods. The spectrum I refer to here is of causality and generalizability. The more precisely you identify a causal relation the less readily it would be generalized and the more general your insight is, the less sure you tend to be about its causal implications. You could contextually tweak a model to incorporate local constraints or do many experiments over and over with small changes or do quasi-causal research or do very large experiments. All are directions research is moving into. But, understanding the extremes is useful.
Friedman or Kremer?
We can choose to ignore some of the circumstances or subjectivity in examining choices of individuals (typically and often stated as assumptions) when building a theory. Milton Friedman’s advice was along these lines. Focus on rational, logically consistent theorizing without questioning the validity of assumptions; to the extent that is instrumental only in improving explanatory power of the theory. If a theory explained the world, the validity of its assumptions mattered much less.
Or, we could go the way Michael Kremer went, rejecting Friedmanian modernism (See here on pg. 476 for his own explanation) in favor of context-based randomized experimentation. (Also enjoy a snippet he cites on the futility of 1:1 maps of the world) In his world, you had to engage with actors directly and chip away at understanding the world one piece of causal relation, one plausible mechanism emanating from one random variation at a time. We really examine one assumption at a time, but what happens when many of the assumptions vary at once? That is difficult to answer. RCTs too struggle with giving a model, a complete story and has problems emanating from exactly the opposite problem of too much context-reliance. That is the Puzzle of Generalizability or external validity, which we shall return to in a bit. Assumptions and context are mirror-images of each other. For now lets ask what from the context can we assume away?
Assuming away
What can we assume away? How do we build actionable theories? Economics keeps going in circles around this. Hayek’s question for example was about whether we could meaningfully distinguish pure logic of action from empirical content of the action in markets? Mises would ask whether a priori logic that can be deduced to rules a.k.a. Praxeology suffices in explaining economies without attention to the content of actions as Mises would argue? Or in wider Weberian terms, the core question was about how practical rationality (undertaking means-ends actions optimally) and theoretical rationality (infer philosophical “meaning” from such actions) were related to each other and to actions themselves?
The assumption of rationality
Understood from an epistemic point of view, economists across intellectual traditions have tried incessantly to assess varying circumstances, rationality and its coherence in institutional and relational macro-design. Each new economic concept then tried to alter or answer an epistemological question adding more nuance to it. Unsettling the prevailing method and giving more epistemic certainty has marked a new shift in coherent understanding of the economy.
Economists often have had better consensus on some fundamental tenets (1, 2, 3, 4, 5, 6, 7, 8) to ensure efficient arrangement of resources. But, how do we diagnose these other aspects that would interrupt our thought flowing from personal rationality to a meaningful theory of the economic world? Being a one-handed economist who studies rational means-ends reasoning to choose sustained efficiency is just hard even when actors are rational. Describing is somewhat easier, analyzing a little worse, examining demonstrable hypotheses a lot more difficult and forecasting is malady plaguing economics. The world is a damned place, complex, if you prefer that.
I have sampled some examples for a survey:
Presuming all individuals act in self-interest (not selfishly but not seeking better for themselves): what is rational for all individuals in their own capacity could be sub-optimal for a group (tragedy of commons, paradox of thrift , prisoners’ dilemma, bank runs, externalities) or for institutions (moral hazard, adverse selection, or information asymmetry more generally, flawed inherent incentive structure, unintended consequences, transaction costs) or for free exchange itself (cronyism, regulatory capture, rent seeking).
Moreover, we will never be able to tell accurately of that person’s circumstances (HANK, In poverty- 1, 2, 3), or a person’s location in society (Institutions, Culture, Social position, Geography), thus failing at anticipating what would be rational for him in that instance. This is if all individuals were rational but to make matters worse, sometimes people just aren’t always all that rational stricto sensu (Bounded Rational, Loss averse, Bad Intuition, Cognitive-Biased, Collectively more rational).
Summarily, telling what people will do, how they will react and how that will change the world is hard. Economists have tried to plug the problems listed above by pulling newer pigeons from their hats. But that an interaction of just these twenty five instances sampled above and many others unlisted could often make the process of free mutual exchange erratic, or somewhat incomprehensible should be clear. Do I then declare objective knowledge to be impossible? Ah, boring. If not for anything else, I wouldn’t do that out of sheer metaphysical urge to make the world make sense. Also pause for a moment, and glance at those links again. That is a lot of progress, no?
New economics students like me get arrested by showstopper locked-in vocabulary non-stop. This is outside of the otherwise performative appeasement-linked narrative-mongering that is characteristic of some experts. This is often of a purely academic nature. “Too much nuance, too little (generalizable) content, no contextual appreciation, lack of system-understanding, not grassroots enough, vague model, unrealistic assumptions, uncontrolled unobservables, political economy of this and that, clarion call for interdisciplinarity, policy-relevant, institutional factors, state of technology and knowledge, human capital, power and politics, etc.” are common academic veils of refrain. I have sometimes found these epithets to be lazy. They often mean an epistemic denial that the different disciplinary branches are stuck in. So let’s dig in and start exploring epistemology through examples.
RCTs, Modeling and other things
Economics, with its penchant for measurable duality, chooses to represent the world as a model and context. Logic and everything else. That indeed makes theorisation tractable. Mathematics as the only internally consistent discipline which is a reflection of our mind and a pure theory of relations of relations, serves as an ideal tool. If we had to design an experiment to test the validity of the model, that would require making real conditions coherent with model assumptions. Friedman would politely decline it as impossible. Kremer, would go on to approximate this in Kenya by choosing not to model a priori, a.k.a. RCT. Assumptions in modeling made a priori often don’t hold and can be difficult to test. Likewise reducing all of context to assumptions for logically exclusive tractability is a difficult venture. One clarification here is in order that I am not yet referring to the problem of selection bias, just the context-assumptions-theory relation at the level of a coherent theory.
RCTs
There is a crucial interdependence between what economics calls assumptions and what they call context. If you can’t vary a variable, hold it constant. If you vary it, observe it. In using modeling as the method, the world outside of mathematically disciplined logic is assumed to be a certain way. If we choose RCTs, the context itself is varied in a random-controlled manner. If we were to backtrack and derive a theory from the inferences of the RCT, we would assume away some context yet again. The only real benefit in this matter is that RCT allows the intervention to play out as it would and therefore might have a better chance at a more precise (but perhaps less general) theory. (Often, that too may not be the case )
Though they can’t be used to answer all questions in the discipline, RCTs offer some respite. RCTs can delineate an effect direction and size without even requiring a completely coherent or internally consistent model of logical choice and mechanism of operation. Some Potato-therapy in Mango River Basin led to a 2 SD increase in word-salad test-scores is a fair and complete summary of a RCT. The author need not always take it upon itself to delineate how. She can only hypothesize the how and accept humbly that this might merely be one of the mechanisms. In other words, RCTs do not rely on a full model to reveal effectiveness. RCTs offer the flexibility to combine different mechanisms without having to completely model the scenario and all contingent relations of variables. I could very well bypass a constrained optimisation effort, use more uncertain terms than mathematics and attempt to engage with the muddy world as it is.
Most fruitful RCTs do try to theorize about mechanisms. Albeit, a model perfectly explaining one RCT in one place can sometimes only explain that part of the world at that point in time. Studying mechanisms that can potentially be general is a qualitative extension of the RCT. A theory would generally emerge from multiple RCTs revealing multiple mechanisms inductively woven together. Understanding effective mechanisms requires a lot more of breathing the context and repeated experimentation to even build baseline confidence in the idea. It often requires different observational methods than quantitative surveys. Why something works and how to make something work are questions fundamentally different from whether something works. The former two can't always be reduced to inferential statements. Embedding yourself and breathing the context to reveal the mechanism, uncle Friedman obviously wouldn’t have liked. Other observational methods would steal the objective thunder forming the basis of such knowledge, robbing the purportedly non-causal knowledge of its appropriate epistemic dignity in research.
Models
Models on the other hand by requiring internal consistency in a priori logic, can potentially reveal flaws in reasoning by demonstrating varied possibilities, including some unimagined possibilities. Intermediate steps of a model can also tell stories, as can logical solutions to the model. But models are bound by constrained optimisation and logical tightness and a requirement of internal consistency. They require an expansive set of assumptions to make mathematisation possible.
Cross-roads again. Where do we go? Knowledge, objective, possible? Let's go one step further down.
Mary Morgan wrote an excellent history of modeling in economics. She argues and interestingly, making matters worse for us, that modeling is also in one sense akin to experimentation. She writes:
“..models, then have a stand-alone, autonomous, quality, that enables them to lead a potentially double life for, I argue, models function both as objects to enquire into and as objects to enquire with. That is, they are objects for investigation in their own right, and they help the economist-scientist investigate the real-world economy…
..model reasoning, as a generic activity in economics, typically involves a kind of experiment.”
Mumble jumble? In short, we think about models while we think with models. Using the model and thinking about it must be so tightly sequential that it is almost simultaneous. Then we can experiment while modeling using models. Dani Rodrik is a proponent of this as we will see later. For now lets quickly also highlight the steps involved in modelling.
Three steps of Modelling
Assuming that a model seeks to discipline thinking in a coherent manner and also provide a framework for experiments, an economist has to face three issues, first two about building, latter about testing.
How does the model abstract from the world, in how it chooses the most relevant parts of the world? This is an inevitable compromise to create an epistemic possibility. Too many object-actions might mean an overfitted explanation, too few might mean under-fitted explanations, parsimonious is hard. Too-tight might mean two things; too local or too situation specific, like a RCT. Too-loose might mean less causal accuracy, less tractability. Like Economics sees them everywhere, tradeoffs are inevitable in studying economics too.
How does it organize these facts thus chosen? Two critical aspects of these are what relational form to choose and if modeling agents, what form of action-orientation to presume. The former has been sought to be disciplined in mathematical theory to allow a priori notions, logical consistency, non-circular reasoning, optimisation etc. Action-orientation deliberates about how the agent is presumed to make choices. If rationally, over what time horizon, whether choosing utilities or not, etc. Is he a sociologically determined tabula rasa? The variables and concepts used when doing this have to be pure in definition, homogenous in application, unchanging in time and insulable from other latent ideas to be useful.
How does the model explain historical data about specific economic characteristics? For all models testing hypotheses, the model from 1 and 2 then is used to make predictions about some stylised facts and these predictions are tallied with different observable economic data. This “tallying” is a statistical exercise in itself and includes multiple problems as we will describe later. The findings then are used to improve the model, discredit the data, design new variables, re-interpret theories, choose between alternate theories, question the hitherto axiomatic assumptions, call for rethinking, furthering ideological ends, calling for new data, including more extensive data, recreating the model in many places and times and so on.
Real world → Identified relevant objects→ Ideas → Concepts → Relations between concepts → Logical consistency → Theory → Theoretical predictions
Thus, a theorist will have to identify elements, moving more and more of them towards the actual and empirical. In isolating facts to study the theorist has to
1) Define new variables, 2) Start with a logical underpinning of how people will act and react [rational ordinal utility with assumption of consistency is standard] 3) She has to clearly describe functional forms of how the entities involved will act or give an exactness to inter-relations of facts [often through mathematics] 4) Then these functional forms will be solved to describe some patterns about solutions of these systems as predictions of how outcome is arrived at from the premise 5) Then using some intermediate equations and final equations, we examine if it is consistent with empirical suggestions 6) Experiment with the disciplining, add relevant facts, change the relations and redesign till the theory is consistent with data at that point in time.
Concise, elegant, simple, parsimonious would make a good model. But there are two other questions that we need to ask. Which model do I use where there are many? When to discredit data and when to reject theory? How do we marry models with the context? Looks like art and I am no Picasso. Ah, fuck it. Let’s hold this for a while. We will also see this manifest in RCTs in a later section. But before I bombard you with it, let's add one more bogey to economic theorisation itself. Until now, we have understood modeling by arranging time across a Cartesian space and thinking about context. Some in economics hated that. Time to them was more than that. Let’s meet with them for a while.
II- Time, thoughts and the ghost of Shackle
Time friction, uncertainty and imagination
GLS Shackle studying economics of Keynes under Hayek’s supervision (itself a rare distinction), boldly declared thus:
“Economics is about thoughts. It is therefore a branch or application of epistemics, the theory of thoughts.”
The world varies in all the ways we saw in the previous section and this variation makes employment of a “pure logic of choice” difficult. But this difficulty is worsened by yet another culprit which is critical to epistemic inquiry. Time.Mises writing a few decades ago in his treatise on Human action, summarizes this best:
“Constancy and Rationality are altogether different notions..”
L.V. Mises, Human Action, 1949
Sounds plain. But the statement is of exceptional import as we shall see. What it meant was that interactions and outcomes of rationally outlined behavior could change with time (also with space as we saw earlier). Shackle thinking along similar lines claimed something bolder. “.. science is merely the recognition and description of constancy.” Therein was a paradox. Science operated on constancy, consolidation and recurrence. But economic rationality also had to make space for origination, imagination, expectation and speculation.
As Shackle described:
“Time and Logic are alien to each other.” …. “Novelty is the transformation of existing knowledge, its reinterpretation; in some degree necessarily its denial and refutation. Mathematics can explore the meaning of what is already implicitly stated, of what is already given.”
G.L.S. Shackle, Epistemics and Economics
According to Shackle, time in its reference to the future held in its womb myriad possibilities arising from the ingenuity of human thought and imagination. Every free man was a free thinker and could inspirit the world with novelty and inventiveness. The world as imagined in a timeless all-encompassing equilibrium described through a model, could not anticipate, incorporate or explain these possibilities.
A natural question to ask is what is the realm in which this holds true? Or in other words, to what extent does this limit our inquiry of the world? One thing economists agree upon is that enterprise is an act of creative destruction and in the Schumpeterian tradition, innovation was a critical source of economic activity. Every invention created temporaryprofits. Through enterprise, through technology, by re-engineering the world in new hitherto unimagined ways, its effects were phenomenal but it was hard to predict. In a macro sense, Technology and innovation or enterprise thus for example could drastically change the world. So could social change, and migration and political propaganda.
Subjective expectations
But, Shackle was saying something more fundamental. Each economic agent according to him imagined a future and birthed it into existence. We cannot use logic to foretell all the future possibilities. Writing half a century before him, Frank Knight understood this too. He saw friction as the source of profits. In his Risk, Uncertainty and Profits; he writes:
“The normal economic situation is of this character: The adventurer has an opinion as to the outcome, within more or less narrow limits. He is inclined to make the venture, this opinion is either an expectation of a certain definite gain or a belief in the real probability of a larger one. …. it is correct to treat all instances of economic uncertainty as cases of choice between a smaller reward more confidently and a larger one less confidently anticipated.”
A fully predictable equilibrated economy that any theory of probability could explain away fully, made profits impossible. This is obviously only true in competitive economies. In imperfect economies, efficiency gains from reorganization within the factors could still reap profits and growth. If we attempt to understand Shackle in the light of Knightian frameworks, the difference in their epistemic inferences is that of degree to which they thought uncertainty to play a role. Knight recognised the existence of objective uncertainty that could be calculated through a priori calculations or earlier probability estimations. The other subjective uncertainty could not be grouped into instances because such an instance was high-degree unique, therefore less understood and highly unpredictable.
For Knight in practice when a man made a single-instance choice, it did not matter whether the risk was measurable or not because the application of exact circumstances of previous iterations to this case was difficult. Uncertainty, according to Knight could be managed by consolidating it, specializing who bears it, controlling future for example by codifying behavior into law, improving predicting power, diffusing uncertainty or redirecting activity to less uncertain spheres. If you have been able to follow all I said, you will understand that most economic regulators do one of these things. Overall, balancing dynamism and uncertainty are difficult. Why? Because predicting is goddamn difficult.
Shackle was even more audacious in his claims. For him, the realm of objective probability was much narrower, almost non-existent. Current Value of things itself emanated from people’s subjective expectations and speculation about the future value of the concerned commodity. According to him, price simultaneously co-ordinated subjective valuations and rendered them temporary, objective, public facts. The originality of thought that would emanate could not be captured in rational frameworks because it was based on an imagination of the world. He says:
“…the region of ignorance is as important and as exploitable as that of knowledge. The exploitation of ignorance is called speculation”
Prices emerged from disagreement of speculators about expected future value of the commodity as bulls and bears settled around it. It was the unknown that created speculation and possibility of value. Tomorrow was a “figment”. Expectation was “origination”, and was “undetermined for all we know.” Valuation was expectation and expectation was imagination. Money had to look forward for value. Once everyone agreed which direction the price had to move, the movement would have de facto taken place.”
Denaturing of time
According to Shackle, economists denature time. They take away the most important facet associated with it; that of imagination and invention and represent time as coterminous points within a space. Such an arrangement then makes way for timeless equilibrium theorization and comparative equilibrium analysis. But such theorization necessarily fails when faced with novelty. Time was not to be understood as continuous versus discrete but as limiting imagination itself. Joan Robinson describes what economists do succinctly:
“Time is a device to prevent everything from happening at once.”
We had to use time to allow sequence, to allow actions and reactions which we could contain into logical relations as varying with time. The question that begs us: Can denatured time, which ignores the role of rational indeterminacy allow us to predict the future? In other words, can we build a sufficiently complete theory of how things will turn out in the future using rational frameworks? Maybe not. We couldn’t explain most important shifts in the arc of history before they happened, for example. We expose ourselves to Black Swans, a fun discussion in itself.
Summarily, old wine in a new bottle. Knowledge? Objective? Yes or No? Let’s try again.
III- Probability and Living by the bell curve
The automatic economist answer to the problem we just encountered is, let’s apply probabilities. This allows us to put a number to the uncertainty. To seek to understand its degree. In that case, the second question to ask is when can probability be used to describe the world? Put differently, is Knight right about the existence of objective probabilities?
Can we study historical past or system characteristics to pin a number to the occurrence of events? One clarification here is necessary. Even though it might seem like that, we are not trying to forecast anything yet. This is just to understand whether a framework that we built out of our mind could explain the phenomenon we were trying to study in a narrowly defined social system.
Dual understanding of Probability
Shackle, being the genius he was, provided us a framework. Shackle also gives us an elegant vocabulary to formulate the distinction in uncertainties made by Knight in terms of probabilities. He suggests that probability plays a dual rolein the world. It is both a measure of chance expressed in the nature of the world (objective) and a mode of reasoning the basis of whose assignment is ignorance (subjective). Pardon me for the long-ish quote that follows. But it is an excellent description of ideal conditions for classical probabilistic inference of a modeled stochastic system. Mumble Jumble? It is a theory of when we can assign a number between 0 and 1 to the degree of chance of something happening based on what happened in the past to a particular class of contexts.
Objective Probability systems
Shackle writes:
“Even in the kinds of treatment which we have labeled objective, probability is a mode of thought. It is, in these kinds, an interpretation of the way things seem to happen in a class of contexts. We have been seeking in the foregoing pages to define that class. We have put a gloss, perhaps an audacious one, on a distillation of some established ideas. If the resulting account of the application of probability to physical systems is accepted, probability, in order to be illuminating, requires certain conditions to be fulfilled:
1. The system must be so circumscribed that its performances can necessarily be classified under some list of mutually exclusive and exhaustive headings.
2. Individual instances of performance can vary in detail but not in general mode of occurrence.
3. Such instances are independent of each other, in the sense that knowledge of one such instance throws no light on the character of the next. This stipulation guarantees the system against the occurrence of a train of instances each leading to a successor more extreme, in some characteristic, than itself. That is to say, the system has no capacity to engender a self-reinforcing and explosive process.
4. The system is guaranteed against invisible change of its constitution. To ensure this, we require it to be insulated from impacts from outside itself, and free from any capacity to evolve by some mechanism inherent in its design.
We select and collect these stipulations, from amongst many which suggest themselves, on account of their bearing on the application of probability to human affairs, in high contrast to the milieu of a physical system.”
G. L. S. Shackle, Epistemics and Economics
The implications for objective or classical probability are, 1) Being able to list all possible outcomes independent of each other whose sum total is one, in other words knowing the sample space fully well 2) Limited variation in the occurrence of instance itself 3) No self-explosive or self-reinforcing occurrence 4) System in which probability is being measured should have same general constitution and should not change with time or place from outside or within, even because of the instance to be studied itself.
To cut through the bullshit, objective probability essentially did not apply to unique (see Black Swan events for example) instances. The event had to have occurred in a recognizable system systematically understood as a specialized class of occurrences which had been studied before. The event being studied now must not have been too different from events studied earlier in its mode, existence and operation. The system must not evolve or explode and the independent elements that acted on the event should continue to act on the event with limited known variation allowed.
RCTs and objective probability
It would do us good to understand RCTs in this light. They allowed you to compare quantitatively similar systems on average through random assignment. If you observe closely enough, engaged with the context, you could come up with an exhaustive list of possibilities and the important ones amongst them for that context at least. The limited variation in occurrence of instances was created through controlled release of intervention style and dosage. A non-self-evolving system was ensured by limiting the setting to a partial equilibrium framework and focusing on limited areas of operation. System was guaranteed no outside change because of the controlled environment and the hope was that the self-evolution would be the same in both groups, treatment and control. Cute. Very smart indeed.
In some ways better than the model. Similarly, variations of model parameters at margins could approximate the limited variation. But self-reinforcement and its effects, to be predicted, needed advanced a priori theorisation and extensive counterfactual reasoning. Or you could go terribly wrong, also possible when applying RCTs to a different time-place. Systems would indeed vary and models had to vary with the system under consideration. You could potentially be an ad-hoc experimenter with models. Mary’s exhortation makes sense.
Experiments gave probability an excellent chance, easily a better chance than models perhaps. But, were RCTs fool-proof in this respect of generalizing with the use of objective probabilities? Perhaps not. Where did things go wrong? All developing countries didn’t form a class. Results wouldn't generalize to different places or even at-scale in the same place. Often two villages within the same block are not similar enough to form a class. (If your brain screams decentralize, think about the downside) Exhaustive list of possibilities across contexts is impossible to build. Multiple outcomes do not remain independent of each other, they co-evolve. Individual instances of performance or intervention and implementation details change with the context, they change with the implementer, they change with the people receiving the intervention. Interventions could self-reinforce and absolutely backfire over time. The way the intervention interacted with the context varied by the context. Surprises were abundant. And the last and the most important, the system engaged with the intervention and evolved in ways that are not fully understood making transferring the learning hard. It could for example be capacity building, new expectations, cultural shifts and many others.
Taleb in his description of Black Swan world explains why this happens. While we will come to it some other day, being an economist is hard. What is she to do then?
IV- Crossing the river by feeling the stones
Shackle conceptualized a kaledic equilibrium that lasts a short while before it disintegrates and bursts. Kaledic representation of economy represented a bounded indeterminacy of rival possibilities. He did not believe like Hayek or Mises that the economy had any long run tendency to equilibrate. Lachmann, Shackle’s student writes about this:
“Marshallian markets for individual goods may for a time find their respective equilibria. The economic system never does. What emerges from our reflections is an image of the market as a particular kind of process, a continuous process without beginning or end, propelled by the interaction between the forces of equilibrium and the forces of change. General equilibrium theory only knows the interaction between the former.”
*To be noted Shackle is critical of the existence of Marshallian equilibria too on the ground that it ignores income effects multiple simultaneous markets have.
L. M. Lachmann. From Mises to Shackle: An Essay on Austrian Economics and the Kaleidic Society
On a closer reading one will see that this means, we could understand change in an economy without completely understanding how the cause was interacting with it. There are many directions to extend these into. For example, what this meant for value theory, learning and adaptive expectations, role of rational expectations, stability of markets and how co-ordinating imagination could explain the existence of non-market or regulatory institutions. Shackle has other critiques of economic theory, especially its obsession with quantity as against form, its treatment of exchange value as a unit, scalarisation, reducing incommensurables to common terms, prices as mere conventions, etc. His book warrants reading and discussion in full. But of import to us was the idea that maybe the world changed before it equilibrated meaningfully in ways that made General Equilibrium theorisation very difficult. The information shortage and limitations of logic are more pronounced than we would want to believe. Most economic theory after Shackle disregards his writing as nihilist, but he had a point and he made it well.
What have we done thus far? We did not challenge methodological individualism. We theorized some aspects of rationality with respect to context, we tried to understand the role of rationality with regard to time and made some observations on classical probability in face of time uncertainty and contextual differences. Contextual diversity could not be disregarded in modeling and time was a “denial of the omnipotence of reason.” What is then the role of rational-agent theories? And experimentation and RCTs? How do these continue to be useful? How do we rescue this epistemic description which is not objectively wrong from the charge of nihilism-that everything is subjective with respect to time and place and therefore theorisation is fruitless? This will be our endeavor in the last part.
Marrying empirics and context the RCT way
There are diverse standards to determine what can be considered an empirically valid parametric of hypothesis inference. It mostly comes from statistics (1, 2, 3 ) but could also be different (as in 1, 2, 3, for example) based on diverse methods.
RCTs offer a peek into well-identified causally linked relation-between one set of actions called an intervention onto another set of closed defined variables called outcomes. It has been sought after as a way to cut through an abundantly complex world, dissect it into small parts, study interactions there closely, compare them with another on average similar part and tell how the world changed. This method is deemed cleanly deductive. Quasi-experimental methods seek to do similar inference. (The other several issues are in this book are here, but that is for another day)
But, lets suppose that all RCTs are ideal and give purely causal estimates of specific relations. Once we collate all the results from RCTs, there happen to be multiple issues in what is called generalizability. Generalizability incorporates four distinct aspects: scalability (1, 2) , adaptability-usability based on intrinsic quality of the intervention, external validity due to contextual challenge and building a general theory of change. The associated epistemic challenge is how do we take swaths of RCT and causal research and inductively weave it into a story of what should be, what can be the problems and how to do it. I as a part of a team tried here to weave evidence, experience and data to create a starting point guide for policy discussions. Rachel Glennerster here argues that we use RCTs to understand mechanisms, then employ those mechanisms common across contexts. Then we tailor the intervention around the mechanism to fit the contextual requirements through deliberative processes. This involves an inductive funneling that looks like the following:
RCT literature → Identify clean causally identified relations in research papers → Closely study their estimates and potential issues in design → collate multiple similar papers, or in other words experiment multiple times → identify the most common mechanisms and points of failure → Deliberate on those with experts having experience with this implementation → Reason out a story on effective mechanisms of effectiveness → Support it with mixed-methods observations about the contextual nitty-gritties -> Carefully translate it to policy-suitable, politically palatable idea with an underlying theory of change with examined mechanisms and points of failure.
Let us quickly summarise the problems with doing this.
Problems of being an empiricist
Empiricism begins with an idea that is sought to be tested. The idea could be a factual question or a hypothesis. The object of inquiry in that sense is theory-laden. The empiricist has to collate data in an interpretable form. Nature and structure of data is to be debated. Data could be large text, interviews, voice and audio or numerical. Economics with few notable exceptions collects and uses tabulated data described as relations between variables for each observation. It could vary over space, time and individual instances.
Simply put, the empiricist then uses averages and rules to related averages to find differences, explain trends, and test theoretical statements and their logical implications. The data collection and use methodology is often disputed for reasons from outright fabrication (1, 2), chicanery like p-hacking to methodological issues with sampling, cross-validity, time, conceptual definition(1, 2) and also less understood reasons. It is difficult to classify data into binary bins. Especially data that will be used for counterfactual-conditional type of what-if analysis is very difficult to classify. Suffice it to say that all the answers are in the process of data generation and being able to say something about the quality of data from the data itself is rare, digging out skeletons is a part of an economist’s job.
Summarily, an empiricist would proceed as follows:
Real world → Identified relevant objects→ Ideas → Concepts → Numerical or categorical variable (if deductive) → Survey or accounting instrument → Data collection Process → Numerical data → Studying co-variance in an inferentially sound manner → Causally/ Parametrically sound estimation model → Output values (effect size) for relations, overall outcome, parameter estimates, stylized facts -> A local theory of change.
This is excellent where subject matter being studied remains fairly constant over multiple experiment iterations. Meta-analyses or systematic reviews work better in medicine because there is something concrete to be said about the precise mechanism and a variety of experiments can be collated together. In development, precise determination of mechanism can be hard because multiple moving pieces constitute a critical part of the success of what is being tested, when moving from causally determined pieces to a universal theory and economist’s knowledge of the space in between, his experience becomes crucial. The empirical is the part of the real, there are some surprise elements which can’t be pre-inferred and they can show up very easily in a new and different context. This makes iterative adaptation when using knowledge inevitable. Theorization about the *real* world is *always* incomplete. The literature is therefore a motivating point or the starting point as done here or here. Collecting frequent data to iterate as mentioned by Karthik here, for example furthers this aspect and therefore becomes inevitable for new generation organizations including governments to deliver.
Some epistemic cautions in empirics
Finally, there are a few things to be said about connecting the empirical and the theoretical and using statistics in practice.
The concept in theory has to be closely proximate with the variable being measured in statistics. E.g. Which index is an appropriate measure of price-level? If the concept and variable are very different and only distant proxies of each other, the analysis will be flawed. This divergence can create disputes.
The variable should be measured at the right level of sub-economy and at time near enough to the present so that the measurement itself is granular enough to explain observable differences emerging from underlying characteristics which we understand poorly. E.g. 15 districts produce half of Indian GDP. (NFHS 2019)
Any proxy imputation by estimation must be heavily suspected and scrutinized.
Data is also abstract, it is constructed while being collected, can be and often is interpreted subjectively. It has to be read to mean only what it can mean but it can be used to mean what you want it to mean. The difference is in getting under the skin of data.
Typified data can mean different things than what the type suggests. E.g. Rape counts in Delhi could often also be consensual couples trying to elope their family. It was classified as rape because that is what the FIR said. See this for more such examples. Such misclassification can lead to bad inference and worse theories. Carefully examine what the data is, and especially what it is not, examine the boundaries of the variable and distribution within those boundaries before using that variable for inference.
Data is collected/generated with a purpose orientation. When using data for other than intended purposes, employ it extra carefully. There is a penchant to use “best available data”, but it might not always be advisable for two reasons: Data points become anchors of knowledge and might misguide intuition, data too old, or too different might convey neither average trend, nor movement around the trend nor distribution appropriately. But, we should not rush to reject any data before trying to identify its signal-noise ratio and direction and extent of bias. How to do that is an art.
Null effects or effects indistinguishable from zero do not mean much for the intervention or the model, unfortunately. If sufficiently powered and precisely estimated they show that in the instant case, in the instant sample, some part of the intervention or the model failed and it is not seen to work/explain. That says nothing about the feasibility of the intervention or validity of the model elsewhere.
Precise causality of estimates is extremely difficult and rare and renders a large base of quantitative information useless. Some workarounds are sometimes found. E.g. An excellent narrative analysis provides some reasoning, a qualitative component guides understanding better, etc. Research studies have validity beyond the size and precision of the estimate. Pragmatic economists always use whatever information is available, caveat it abundantly and make sure their peers understand what is being done.
Statistical validity and procedural sanctity for any quantitative estimate are critical to study. Qualitative studies either stand-alone or done as a part of quantitative studies of whatever rigor can provide a better grounding for overall complete comprehension.
The question we ignored when trying to build a theory; i.e. about the fundamental nature of the content of the model, e.g. whether the model is representative agent; becomes important at the last step of this theory where we attempt to backtrack theoretical inference to the world if the data fits the predictions of the model.
Inferential empiricism is important but so are descriptives. They can give us important facts to build our story on. Given the challenge associated with making empirical sense of the world and then projecting it onto the actual world, we are somewhat successful, but now always. What we understand from the empirical and what we can say about the actual are distinctly different. I propose that the empirical and actual world are different spheres with related but different ontological properties. In the next section I wish to substantiate the difference in ontological properties of the actual and the real world. What are these differences? Why do they emerge?
Institutions, context, details and applying theory
The world is second best, at best. -Unattributed.
We do not understand what will become of a theory in a new context. A new context might absolutely mangle our theory and birth consequences we never intended. There is a constant back and forth between models and data. We examine our theory and the world simultaneously like Mary would suggest. When approaching a problem as a practitioner, the following become crucial:
Understanding the empirical background in which one is to operate thoroughly.
Identifying the relevant facts and acknowledging theory ladenness of observation.
Choosing the most parsimonious theory that is sufficiently lean but still has meat to stand by itself.
Validating multiple theories statistically. Identifying the most critical assumptions based on which the application of the theories hinges.
Double checking whether those factual assumptions are statistically validated.
Listing rival hypotheses with subjective probabilities so that the decision maker pays attention even if the list is not omni-competent.
Now, for understanding the empirical background and identifying the relevant analogues of assumptions in the world of operation, an intimate familiarity with that world becomes crucial. This is often also called the institutional setting or cultural background. It could refer to people’s outlook to any of the following in that setting for example:
Participation in economy and society, engagement with issues- extent and manner, notions of morality (fairness, dominance, unkindness, loyalty, sanctity), formality of operation-contracting, branding, risk-attitude, presence and nature of hierarchies, creativity, professionalism, specialism, embeddedness- ingroup-, friction to foreign-ness and institutions within, norm-consistent behavior, interpersonal relations- kinship, social organization, attitude to labor, leisure and consumption, attitude to loss and compensation, violence, norms about expression of basic and higher cognition emotions, conviviality, relation with material-life and infrastructure objects, process of adaptation and evolution of norms, treatment of time, path-dependence, austerity, habits, rules and laws, etc.
Each of these is independently also an object of economic inquiry, but I mean these only in a general intuitive sense. We also scantly understand how these exactly interact to produce a system. All these characteristics apply to both human participants but also institutions. Institutions can through conscious self-reflection evolve to have their own characteristics. They might resist or enable change, be entrepreneurial or not, they might be inspirited or dud, they might prioritize merit or relations, they might have clearly defined goals and methods, they might be top-down or bottoms-up, they might have process-orientation or outcome-orientation. They might be person-centric or value-centric. Participation might be costly or free. It might moreover be easy or difficult to create and sustain institutions. They might break with size. We can also use economics to further break these down and understand the behaviors of actors as composite parts of these institutions themselves, but systems themselves have characteristic traits which need to be understood. But, Historicity is important. Language (1, 2) can be critical because for example it can explain how and how far a particular phenomena penetrate the conscience of that society, it might also define accessibility.
Context involves more than institutions, people and culture. It involves geography and design too for example. Is it better to be in plains or coasts? How much does it rain? What crops can I grow? How deep do I need to dig for water? What's the slope of land like? Does the river water carry Hepatitis C virus? Does my business depend on parking space? Walkable promenades better? Floods? How about forests and related rights? How is the population arranged (For marathi readers interested in rural Maharashtra)? What is the historical distribution of things of value? What if there are too many cyclones? Which Gods do they worship and what festivals?
The more local we go, the less valuable averages are and the more salient these things become. Everything becomes everything. Settings differ and therefore they matter. One last point of import flows from this. Each economy has multiple embedded sub-economies. At which level we choose to study them matters. How those levels relate to each other matters too. Politics, trade, migration, public finance, monetary integration all determine the extent of integration and difference from other sub-economies. This gap between the real and the actual is perceptible but not fully comprehensible. I thus think a shift to a new paradigm in economic epistemology explicitly accepting these differences can be useful.
V. Critical realist Economics
The need for modeling arises from an epistemic angst. The observable and empirical does not exhaust ontology. Ontology of social experience underlying the economic endeavor becomes manifest, often rendering the economic empiricism an incomplete exercise. Humans transform social reality through continuous action on poorly understood or fully understood norms and in the process the norms themselves evolve. The need to build a demonstrable, consistent within our mind and therefore logical framework to explain economic phenomena is at odds with what our mind can understand about the world. This view is most commonly understood as critical realism. It is a departure on a skeptic or relativist worldview which would deny the possibility of knowledge. Critical realists understand the empirical to be a subset of the real and understandable world.
Source: Somewhere on the internet, Thanks.
Critical realism espouses a layered understanding of the world. It takes the empirical to be a part of the actual world where mechanisms operate. The empirical is distinct from the actual because of our epistemic methods and their limitations. In the actual world observed mechanisms and their effects come into action. The real world on the other hand comprises deep lying structures and relations that birth social phenomena and events. These structures and their interrelation is mostly unobservable and can be penetrated only by a priori logic.
Critical realism is precisely between postmodernism skepticism in epistemology and a modernist positivism in methodology. Critical realists agree that there is an objective reality but the observable world that can be empirically studied is only a small part of the real world which remains impenetrable by direct perception. We can only conjecture about the happenings in the real world by understanding the empirical facts and their relations. Critical realists engage with four ideas seriously:
A close attention to social ontology
Methodological Pluralism
Reflexivity on part of the researcher
Use of Retroduction (Testable educated guessing) along with induction and deduction
Focus on explaining generative mechanisms of causal relationships as against conjunctive relations themselves
It differs from holism in its lesser emphasis on interconnectedness of the world. As such it allows better focus on identifying individual mechanisms in the empirical and actual world rather than trying to study grand emergent ontological properties. Our understanding of the world is enriched by plural methods to attempt to reveal the nature of ontology. It rejects reductionism into small parts. Critical realism also helps us avoid the fallacy of composition by distinguishing the actual from the real. It is a humble starting point that starts with ontology and structures methodology and epistemology to respond to the needs of such an ontology.
Economics because of its early modern beginnings, victorian determinism and the post-structural contextualisation and constant search for empiricism, periodic renewal of vocabulary, value-neutrality outside of the singular utilitarian framework, grounding in mathematics as a way to impress our mind onto concepts, resultant engagement only with scalarity, early incorporation of and fundamental contribution to statistical thinking, functionality as an instrument to efficiently fund war and given an external push mainly from government to constantly search for difficult answers to crises has staunchly remained realist. It always acknowledged the possibility of objective knowledge and resisted relegation to relativism or skepticism, which in part explains its success. Critical realism embeds this positive modernist methodology into an epistemic humility. Critical realism can truly enable us to cross the river by feeling the stones. This becomes especially relevant in a data-scare world. This is a very valuable guide, especially for applied economics and policy relevant research. It holds substantial promise as economics continues to ask bigger questions and answer them with finer tools, multiple iterations and a wide variety of models. Economics, especially applied microeconomics has also unknowingly walked into the domain of critical realist understanding as was seen in all that we tried to understand in this article.
Critical realism in practice
For practitioners, Dani Rodrik, writing in Economic Rules (see chapter 2) provides a good guide to a part of this. His approach to economic theory is a compromise between Friedman and Kremer.
Use many models and often a combination of models -> Identify the critical assumptions -> Match those and the primary factual premises and direct and incidental logical implications of the model to the context- > Choose the appropriate model(s).
More models, the merrier. The more mechanisms we understand and know to work the better. Rodrik proposes a method to undertake diagnostics of model suitability too. Rodrik also argues that RCT results should also be demonstrated as generalisable models by abstracting away some of the context. When we have modified the model and identified the best match we can also use multiple models to explain phenomena. (See here) Rajan argued for us to assume hobbesian anarchy and study institutions. Samrudha wrote an excellent post about it. This too can be fit in a critical realist endeavor.
An explanation thus arrived at will then be iterated in the world of action, feedback sought and intervention tweaked and updated. That is because our understanding of the actual world is incomplete and that of the real world is nil.
Summary policy guidance: Generate alternatives, select the best, focus on co-evolution of the intervention with systemic needs assessed from time to time.
Insulate area of study and its relation with external world+ Institutions+ factual premise+ Modelled rational behavior(s)/ theories+ contextual insight+ feedback mechanisms+ Problem driven iteration + [Discussion and debate + generating new data].. In loops.
This reminds me of Keynes writing about the ideal economist, he says:
“The master-economist must possess a rare combination of gifts .... He must be mathematician, historian, statesman, philosopher—in some degree. He must understand symbols and speak in words. He must contemplate the particular, in terms of the general, and touch abstract and concrete in the same flight of thought.”
Become a process-thinker, context-thinker, institution-thinker, change-thinker, rational-thinker. Master rules and processes of rule formation and social dynamism. Humility could be a remarkable trait in economists given how much we are ignorant about, a critical realist perspective can help. If we end up believing that we know more than they do, we come close to what Taleb calls a platonic fold.
Like Shackle said: “An economist is a sailor. His model are the gridlines on the map.” I add that data are his compass, concepts, boundaries, past and peers keep the ship guided and afloat. This ship though lives in the imagination of people, it sails, sinks and changes destination in that imagination. Embrace critical realism.
I can know economics, but my knowing is useful only insofar as readers like you make it to the end of this post and tell me all ways in which I am wrong. Thank you for reading.