---
url: "https://youtube.com/watch?v=31GUkCBD-Uc&is=A1yYV6cfrDa5XMW1"
title: Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber
source_kind: youtube
author: AI Engineer
captured: "2026-08-09T18:08:10+00:00"
comment_tree: false
topics: [other]
summary: Uber engineers discuss building closed-loop evals for a multimodal agent that enhances food photos at scale, balancing authenticity, quality, and merchant brand preservation.
status: ok
---

# Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

Channel: AI Engineer

## Transcript

[1s] [music]
[17s] >> My name is Jay and I'm here with Sonya.
[19s] We are part of the computer vision team
[22s] at Aruba. We're going to talk to you
[24s] about a real world production use.
[27s] Oh, my son done. Okay.
[30s] Try again.
[32s] Okay. Don't worry. I'll I'll manage. You
[34s] hear me now?
[36s] Okay, so we're going to talk to you
[37s] today about a real world production use
[40s] case
[41s] and specifically we're going to dive
[43s] into how we design the e-bows and the
[45s] e-bow loops. So
[54s] All right, cool. So just before we get
[56s] into the agent design,
[58s] we're going to talk about a little bit
[60s] about the use case. So our delivery
[62s] marketplace Uber Eats, we do about 90
[64s] billion
[66s] run rate per year at the moment.
[68s] We were adding millions of items to the
[71s] marketplace each and every year.
[74s] Sorry, every every month. We're growing
[76s] at 20% year-on-year and and we operate
[79s] in 10,000 cities globally. So not many
[81s] people actually know this but our
[83s] delivery marketplace is just as big as
[86s] the mobility side on Uber today.
[90s] Visual content actually plays a really
[92s] important role for the user experience.
[95s] So a photo is quite often the first
[98s] signal that a customer gets that gives
[101s] them that initial impression about a
[103s] merchant.
[104s] So a good photo can make the difference
[107s] between someone scrolling through the
[108s] feed and actually clicking on an item
[110s] and adding to the cart. And more and
[112s] more we're seeing different modalities
[114s] on Uber Eats uh especially video
[117s] content.
[119s] But this is a problem.
[121s] So, our smaller independent merchants
[123s] simply just don't have the level of
[125s] quality for their photos that reflect
[128s] what the eater is actually going to get.
[131s] And when we speak to our merchants,
[133s] there are three themes that kind of
[134s] emerge.
[135s] Lack of time,
[137s] lack of know-how, and costs cuz these
[140s] professional um photo shoots actually
[142s] cost a lot of money.
[144s] And this can be especially [snorts]
[145s] problematic if the merchant is updating
[148s] their menu over time.
[152s] So, this problem is actually pretty
[154s] challenging to solve for at scale,
[157s] right? Because our consumers, they want
[158s] authentic, real-looking photos,
[161s] um but a meaningful fraction of uh
[164s] consumers actually distrust anything
[166s] that is AI-generated. So, if you open up
[169s] the Uber Eats app, the last thing that
[170s] you want is to be scrolling through uh
[173s] you know, food photography that looks
[174s] like AI's lock.
[176s] So, we're threading the needle here. We
[178s] need to be able to stay faithful to the
[180s] original image, preserve the brand of
[183s] the merchant, and avoid everything
[185s] looking the same. If we have the same
[187s] prompt for every photo that we're
[188s] editing, the diversity of the
[190s] marketplace is going to collapse.
[195s] We also, because we operate globally, we
[198s] also have this long-tail distribution of
[200s] different quality that we see it across
[202s] the marketplace.
[204s] Um so, we've got some examples here. You
[206s] might see food photography that, you
[208s] know, has poor sharpness, poor
[210s] composition, not centered, uh or or poor
[213s] colors as well. We also have a wide
[216s] range of spectrum of user-generated
[217s] content on the platform as well.
[222s] So, what are our goals when we're
[223s] designing these agents? When you think
[225s] through these goals, you might actually
[226s] be thinking through, you know, your own
[228s] agents that you're building yourself.
[230s] But for us, it's about one, preserving
[232s] authenticity and trust.
[234s] Two, improving the quality when we need
[236s] to. So, we want to be able to improve
[238s] quality selectively.
[240s] We want to optimize globally for for the
[242s] entire marketplace. We don't
[244s] cannibalize certain merchants. We want
[247s] to ship safely, and this is going to be
[249s] an important theme throughout the talk.
[251s] We want to learn continuously,
[253s] and we want to operate at scale in a
[255s] cost-efficient manner.
[258s] So, agents are actually really
[260s] well-suited to solve this problem.
[263s] So, if you imagine a spectrum, on the
[264s] one side, you've got something that's
[267s] more deterministic. It's more
[268s] rules-based. Um and uh you you have more
[272s] control over it, but it's fairly it's a
[274s] brittle system. It's not actually going
[276s] to be able to scale for the entire
[278s] marketplace.
[280s] Imagine the other side, you provide an
[281s] agent with obviously a lot of
[283s] creativity, it has a lot of agency. Um
[286s] and that's actually what we want to lean
[288s] into,
[289s] but we can't leave that unconstrained,
[291s] right? Cuz we have certain safety and
[293s] certain guardrails in place that we need
[295s] to adhere to.
[296s] So, we want to find a balancing act.
[299s] Uh and that's kind of set the principle
[301s] for the way that we think and design
[303s] around agents and evals.
[305s] So, now we're going to actually like
[306s] dive a little bit deeper into a
[309s] simplified but representative example of
[312s] what we have in production.
[314s] And we're going to go through each stage
[316s] and how we eval it, and then talk
[317s] through some continuous learning loops
[319s] as well.
[321s] So, first up, we have what we call an
[323s] image understanding and routing agents.
[326s] So, this is where multimodality is is
[328s] pretty important. We actually ask the
[330s] LLM to describe what it sees in the
[332s] photo.
[333s] Um and then we we create a structured
[335s] output from that, and we send it to a
[337s] router.
[339s] The router will then determine, do we
[340s] enhance it, or do we skip it? We skip
[342s] it, we will keep the original.
[345s] If we enhance it, we send it to our next
[347s] agent,
[348s] which is an image editing agent. And
[350s] this can actually run in a loop. So, it
[353s] gets feedback from a QA agent. Um it can
[356s] edit uh
[358s] in the in this loop and self-correct and
[360s] fix things
[362s] um as it goes.
[364s] If it goes through a number of loops and
[366s] it still fails, we we don't publish it.
[369s] Then we actually send it to a final
[370s] post-processing and QA step.
[374s] If that's all good, we'll publish it to
[376s] the menu.
[377s] And the last thing that's really
[379s] critical is we log everything.
[382s] Just a quick note about logging.
[385s] Don't know if you can actually read the
[387s] JSON here, but you might notice that all
[389s] of the agents in this end-to-end
[391s] orchestration is within one It's It's
[394s] basically a flat structure in this JSON.
[397s] Um and so, this is actually incredibly
[400s] useful for the entire team
[402s] because anyone, be it non-technical uh
[404s] technical um folks on engineering
[406s] product, can actually dive in um and
[409s] look at specific cases to diagnose and
[411s] also roll up things to look in
[413s] aggregates.
[414s] Um and it's important to note here that,
[416s] you know, we think this is important to
[418s] start with. You want to start with your
[421s] logging cuz if you don't start with it,
[423s] you have nothing to optimize for, let
[425s] alone set up a self-learning loop. And
[427s] at Uber, we um we use our eyes.
[431s] Cool. We're going to dive um a bit
[433s] deeper into the router.
[436s] So, the router's actually pretty
[437s] straightforward. If you remember, we you
[439s] know, we have this multimodality input.
[441s] We look at certain text description
[443s] metadata, the image itself. We ask it to
[445s] to to to under um describe what it's
[448s] seeing. We create structured output from
[450s] that. With that structured output, we
[452s] can then grade against a rubric. So, we
[455s] have these pass and fail criteria. The
[457s] last step is we want to decide whether
[459s] or not we should enhance or skip.
[463s] How do we actually eval this?
[465s] This is you could think of this as a
[466s] more sort of traditional classifier. So,
[469s] here we we have a confusion matrix. You
[471s] know, many of you are probably pretty
[473s] familiar with this.
[474s] Um but we can look at things like the
[476s] true positive cases, the false negative
[478s] negative cases, and so on and so forth.
[479s] Essentially, what we're doing is we're
[481s] measuring the precision recall.
[483s] In practice, your routers might actually
[485s] be much more sophisticated. So, for
[487s] example, we might want to route an image
[490s] to a lower latency smaller model to be
[492s] able to save on cost and improve the
[494s] user experience at the trade-off of
[496s] quality.
[497s] And if that's the case, instead of
[499s] having a 2 by 2 matrix for your
[501s] confusion matrix, you might actually
[502s] have an n by n matrix.
[505s] Where each grid is actually telling you
[508s] whether or not you're correctly routing
[510s] to that specific branch.
[512s] So, I'm going to now hand over to Somya
[515s] who's going to dive a little bit deeper
[516s] into how we handle drift and human
[518s] alignment.
[528s] >> So, now that we spoke about how we eval
[530s] the routing, I want to talk about how do
[532s] you get the first version of the model
[533s] out.
[534s] For our use case, we consider human
[536s] labels as the golden source of truth.
[538s] And this is what we want to align our
[540s] models to.
[541s] The way we do about this is we go
[543s] collect a dataset which is
[544s] representative. So, you know, different
[546s] cuts, geographies, dish type, image
[548s] quality type. Send it to our human
[550s] labelers and give them a very objective
[552s] guideline to label on.
[554s] This is to remove any subjective biases
[556s] or any noise coming in from human
[557s] labelers.
[558s] Once we've got that system set up is
[560s] when we start tuning our model. We take
[562s] our agent, we go ahead get output from
[564s] the agent, compare it to your golden
[566s] dataset, evaluate if it's good enough to
[568s] ship, if it meets your guardrail
[569s] metrics, you go ahead and ship it. If
[571s] not, then you go tune and you keep doing
[573s] this until you meet your guardrail
[574s] metrics.
[576s] For routing, our guardrail metric is
[577s] recall. We don't want any bad image to
[580s] slip through our system.
[584s] Here are some examples of the failures
[585s] we've seen.
[587s] Uh on your left you see a very good
[588s] image of cheeseburger. Uh on the right
[591s] you notice that the routing agent
[592s] actually failed this. It said the
[593s] technical is low ball and it will go
[595s] send this image for enhancement. Now
[597s] there's two challenges when you send
[599s] this image for enhancement. Firstly, you
[601s] pay the compute cost for a zero quality
[603s] lift from this image. And secondly, uh
[606s] there is a risk of degrading this image
[608s] given it's already such a high quality
[609s] image.
[613s] And on the other end of the spectrum,
[614s] you have a recall miss. So on your left
[616s] you have an image with six chicken wings
[619s] and on your right if you notice the dish
[620s] name, it says eight pieces chicken
[622s] wings.
[623s] And your routing agent approved this
[625s] image. That means So now there's a risk
[627s] here if you send up send this image for
[629s] enhancement and you only see six chicken
[631s] wings, there's a chance your model's
[633s] going to hallucinate these two extra
[634s] wings to to match the description.
[636s] And that's also an that's a the
[639s] cut we take at our faithfulness metric
[641s] that Jay earlier showed us.
[643s] So the meta point I'm trying to get here
[645s] is you've trained your offline model,
[646s] but there will be long cases where your
[648s] model is going to continue to fail and
[650s] the static model will not work in the
[651s] real system. You need a way such that
[653s] your prompts, agents, system itself is
[655s] evolving over time.
[657s] And that's what we've done
[659s] uh for our system as well. And I'm
[661s] talking more from the routing
[662s] perspective, but every component in our
[664s] system is able to tune itself uh for any
[666s] drift online.
[668s] So what we do is we sample production
[670s] data at regular cadence,
[672s] uh send this to the human labelers with
[673s] the same guidelines that we have seen
[675s] before. Once you've got that data, we
[677s] compare our agents' output with the
[679s] output we got from the labelers and see
[681s] if there's a mismatch. If there's a
[683s] mismatch, we have an umbrella diagnosis
[685s] agent which takes in the feedback,
[687s] localizes where this issue is happening,
[689s] and and and triggers our auto-tuning
[691s] pipeline.
[693s] Once we tune this agent, we go and
[694s] benchmark it against our golden data set
[696s] that we saw earlier, and if we pass our
[698s] golden data set on the metrics that we
[700s] had designed, we go ahead and ship this
[702s] model.
[703s] Uh if not, then you kind of keep
[704s] iterating. And this happens on a regular
[706s] basis on production data set.
[708s] Um the beauty of this is this is
[710s] completely config driven and doesn't
[712s] require human in the loop. Your
[713s] diagnoser agent can write your config
[716s] and trigger the auto-tuning pipeline
[717s] here. And this is what will keep your
[719s] model sharp over time. You will have one
[721s] static model with the offline, but this
[723s] is what is going to keep your system
[724s] alive.
[728s] Um so Jay is going to spend more time on
[730s] the diagnosis side of it. What I want to
[732s] do is zoom into the auto-tuning bit. And
[735s] again, we're looking at routing, but
[736s] this is how we tune every agent in our
[738s] system.
[739s] Uh so we start with a target agent, and
[742s] we've already got these uh unseen eval
[744s] samples from our humans.
[745s] We go find out the mismatch and matches
[747s] and call a prompt optimizer agent. Now,
[750s] this itself is two sub-agents. There's
[752s] the reflect agent and the up synthesize
[754s] agent. What reflect does is it it just
[757s] looks at the mismatches, tries to find
[759s] remove any noise, find any systemic
[761s] issues that might be in your data set,
[763s] and
[764s] reflect on it and send that feedback to
[766s] the synthesize agent. Now, the
[768s] synthesize agent takes this feedback. It
[770s] has your agent config. It goes and
[772s] updates your agent with the new config
[774s] based on the feedback it's getting. And
[776s] goes and benchmarks again. If this
[777s] benchmark is passed, you actually
[779s] register this new agent in the new agent
[781s] store. And next time your production
[783s] runs, you pick up the new version of the
[785s] agent.
[787s] And this is a closed-loop system as I
[788s] mentioned, no human in the loop. We
[790s] definitely have observability on the
[792s] guardrails, quick rollback built in in
[794s] case of any issues with the system
[796s] itself.
[800s] Moving on to the next step of our
[802s] orchestration flow. So we spoke about
[803s] routing, moving on to the enhancement
[805s] bit of it. It's a three-step process.
[808s] What we do is the first step, we
[809s] generate a prompt specific to this
[811s] image. We take in the description, we
[813s] take in the directives we were getting
[814s] from our routing agent, and we go ahead
[816s] and generate a prompt for this image.
[818s] What needs improvement in this image
[819s] specifically? And we go ahead and
[821s] enhance this image. Then you've got the
[823s] QA gate, which is a multi-dimensional
[825s] gate, looks at multiple things like
[826s] plating, faithfulness, colors. And if it
[829s] passes is when you actually go ahead and
[831s] publish this. If it doesn't pass, you
[833s] take the feedback back from the QA gate,
[835s] push it back to your generate prompt
[836s] along with the initial inputs you sent
[837s] it, and go ahead and enhance it again.
[840s] So, there's two end results here. You
[841s] either keep enhancing for K iterations
[843s] and you pass your QA gate and you
[844s] publish, or you take a coverage hit and
[846s] you never enhance this image.
[851s] Here's an example. On your left, you see
[853s] a bowl of sweet potato fries. We send it
[855s] up for the first iteration and our QA
[856s] agent rejects it because the portion
[858s] size is incorrect, the plating is very
[861s] unrealistic. We take that feedback in,
[863s] go for the second iteration, and we're
[865s] actually able to pass it the second
[866s] iteration. So, the metric we are
[867s] measuring here is pass at K. Pass at K
[870s] is essentially the pass rate at Kth
[872s] iteration. And ideally with the more the
[874s] iterations, your pass rate will increase
[877s] because you're getting more feedback in.
[880s] Now, I'll pass it on back to Jay to
[882s] cover the rest of this.
[887s] >> Thanks Thanks, Somya.
[889s] Um
[890s] So, yeah, just before we end here on the
[893s] um on on the generation of the vowels,
[896s] we use what's called pairwise
[898s] comparison,
[899s] right, for our pass at K. So, it's
[901s] looking at the input image and the the
[903s] output image, and it's assessing whether
[905s] or not it's better.
[907s] But how do we actually find what's
[909s] better? So,
[911s] um we're not going to dive into too much
[913s] of the details here cuz this is kind of
[914s] like proprietary stuff, and so we'll
[916s] just mention it at a higher level that
[918s] this is where you sort of For least for
[920s] us at Rue Ba, we have to make sure that
[922s] we're aligning with product, design,
[924s] policy, legal. And this is where we're
[927s] baking in what we define as a better
[929s] image on the platform into our Evals.
[933s] Um so, examples here, is it faithful? Is
[935s] it complete? Is it natural? Is it
[937s] realistic? And there's a bunch of other
[939s] things as well. The output of this is
[941s] then uh a yes, no, or unsure.
[945s] So, here are some examples of failure
[947s] modes.
[948s] So, input and output on the right. The
[950s] inputs on the left, outputs on the
[952s] right-hand side. This might be a little
[954s] bit uh difficult to to see at the first
[956s] pass. We actually added shrimp here and
[958s] we shouldn't be. So, we failed
[960s] faithfulness.
[963s] This is where we go the other way.
[965s] So, the input um
[967s] has some source at the bottom of the
[968s] sushi. We actually remove it.
[971s] So, we failed completeness.
[975s] Here's actually a pretty interesting
[976s] example where the agent actually
[979s] attempted a more creative edit in the
[981s] first iteration.
[982s] Um and then the QA said, "Nope, it's not
[984s] good enough."
[986s] Uh and then it actually oversteers the
[988s] other other way.
[989s] And it becomes overly conservative.
[992s] Sort of falls back to this generic
[994s] ceramic plate uh ceramic bowl, sorry.
[997s] So, this is an example of a reward
[998s] hacking actually. And and this is a
[1000s] nugatory change, but something that we
[1002s] don't think is a meaningful or
[1004s] influential change despite the actual
[1006s] raw pixels of the input and output being
[1008s] pretty different.
[1010s] Here's another example where in the
[1012s] output the plate is covering the sauce.
[1015s] This is an example where for some some
[1018s] of the frontier models that we're using
[1019s] for the actual image editing, some of
[1021s] their um
[1022s] some of their problems will actually
[1023s] sort of leak up into our applied use
[1025s] case.
[1026s] Um and so, so object coherence and
[1029s] physics plausibility of the Evals that
[1031s] sometimes will coordinate with the
[1032s] frontier teams and and let them know
[1034s] about these problems and work together
[1035s] with them.
[1038s] Here's uh an example of why
[1040s] multimodality is is pretty important. In
[1042s] the input and the output, we we can't
[1044s] actually see that there are eight pieces
[1046s] here of of the wontons.
[1048s] So, we're not confident, actually. We're
[1050s] not sure. And so, this is an example
[1052s] where we would actually reject it in
[1053s] production and and it wouldn't it
[1055s] wouldn't go through.
[1059s] So, the last step after all of that is a
[1062s] post-processing and what we refer to as
[1065s] the publish-ready QA. This is the final
[1067s] gate before we decide we want to publish
[1070s] something to production.
[1073s] Here, we do some policy checks.
[1076s] We also do some more quality checks.
[1078s] Um and you might be wondering, like,
[1080s] we've already done some QA. Like, why
[1082s] are we going to do another step of QA?
[1084s] The reason is because we think of this
[1086s] like a Swiss cheese model.
[1089s] So, we want to try and optimize for
[1091s] reducing the chance of a failure getting
[1094s] into production. And so, there is some
[1096s] redundancy here or there.
[1098s] And that's okay.
[1100s] Um and so, this QA gate is is a little
[1103s] bit more holistic. It captures more
[1104s] things. But, it also will will try and
[1107s] flag things that we should have caught
[1108s] upstream, as well.
[1112s] All right. So, we've talked about
[1115s] a couple of uh
[1116s] feedback loops here. So, to summarize,
[1119s] we talked about predominantly this first
[1121s] one here, which is the model loop. And
[1124s] this is accounting for drifts and
[1126s] aligning with human labeled data set
[1129s] that we have and we've established
[1131s] offline.
[1132s] But, we actually have more feedback
[1133s] loops.
[1135s] So, we we have that Uber what we we have
[1137s] is a is a great sort of dog dog feeding
[1139s] culture.
[1140s] Um we will test apps before they go
[1142s] live. Um but, we also have when it goes
[1144s] live in production, how do we get that
[1147s] feedback back into our agent to be able
[1149s] to steer it appropriately?
[1151s] So, as we're adding more of these
[1152s] feedback loops, we want to be able to
[1155s] generalize the system.
[1157s] So, this is where we've actually created
[1159s] um a higher level of abstraction on top,
[1162s] which we call the diagnoser.
[1164s] So, the diagnoser can take in any input
[1166s] from these different feedback loops that
[1168s] we're we're capturing. It can reflect on
[1171s] what actual agent within the overall
[1173s] system needs to be optimized, and it can
[1176s] route that agent to be able to fix that
[1178s] configuration specifically. It could be
[1180s] one agent, it could be multiple agents.
[1185s] So, here's an example of internal dog
[1187s] fooding. You might see these in sort of
[1189s] different apps that you've got where you
[1190s] got the thumbs down and the thumbs up.
[1192s] We also take some free form feedback as
[1194s] well.
[1195s] Uh and this is actually great cuz we'll
[1197s] get feedback from merchants directly.
[1199s] We'll get feedback from, you know,
[1201s] design teams, other product teams uh at
[1203s] Uber. And we'll incorporate that
[1205s] feedback back into our diagnosis step
[1207s] and tune the system over time.
[1211s] Again, similar sort of workflow pattern
[1213s] here. We'll replay the examples that we
[1215s] know are those ones that have been
[1217s] flagged, be it good examples, be it bad
[1220s] examples, uh and then we'll benchmark
[1222s] the metrics before we push the latest
[1224s] config version.
[1227s] The last step is is actually getting
[1229s] this into production.
[1231s] And and this is where we're looking for
[1233s] a whole heap of different metrics we
[1235s] track for for the marketplace quality
[1237s] and health. Uh I've just called out one
[1239s] here, which is conversion. So, we're
[1240s] looking for improvements in people
[1243s] adding to cart, converting, completing
[1245s] their orders.
[1246s] Um I think this one's actually an
[1248s] interesting one to call out because now
[1250s] at I mean, at least at Uber, but
[1251s] especially in production um settings at
[1253s] scale, you have a wide um
[1256s] uh
[1257s] you have a lot of data that you can
[1259s] actually slice and dice.
[1260s] So, in this area as opposed to the
[1262s] others, what we can do is sort of slice
[1264s] by geos, by device type, by dish type,
[1267s] etc. And we can look at where things are
[1269s] improving in different segments and
[1271s] actually tune on certain segments as
[1273s] well.
[1277s] Cool, and that's it for our
[1278s] presentation. Appreciate it.
[1280s] >> [applause]
[1296s] [music]