This follows the previous post on Agile v. Waterfall in Optimization Solution Development.
Why don’t all optimization practitioners use Agile? What’s the catch? Let’s consider the challenges and how to overcome them.
Decomposing the Model
For traditional software development projects like creating a website, Agile is straightforward: you might one week build a new form, then a new page, then another page, and so on. It is evident how to decompose the application.
In optimization, breaking down a model into small, manageable iterations can be difficult. Components of the model are often interdependent, and changes in one part can affect the entire system. To embrace Agile, optimizers have to overcome the fear that the users will outright reject their solution and kill the project if it isn’t perfect or almost perfect when they first see it.
While Agile focuses on short-term iterations, optimization models often require a long-term view to ensure that the final model is robust and effective. In early stages, the model might only deliver partial or suboptimal solutions. How do you decompose a model so that users can provide feedback as you iterate and progress according to your roadmap?
Determine the subset that makes enough sense to the users. For example, we had a project that entailed optimizing the flight crew for a fractional jet company. There were multiple stages to their crew planning, so we started with the “day ahead” which optimized the deployment of the jets. We then built a model that repositioned the jets to be ready for what was planned the day ahead. Lastly, we built a “service recovery” model that optimized responses for disruptions like mechanical failure and weather. Through this iterative development, we presented to our clients defined subsets that were each sufficient to be used, rather than attempt to deliver the entire solution at once.
Here are several examples of decomposition based on other client projects:
- For a client biotech manufacturer, we first built a model that optimized equipment scheduling. Then we added personnel scheduling, and then we added functionality related to quality assurance and shipping.
- For a bulk chemical transportation company, we started by grouping orders together and making good tours. Then we optimized assignment.
- For a Class I railroad, we are working in a railyard, where there are continual operational decisions. We decompose the complex problem into subsets such as: where a train will be landed, where it will be outbound, track assignment, train composition or “consist”, and switching from receiving to forwarding. In prioritizing these subproblems, there are rules related to safety and other considerations.
Incrementally Adding Complexity
It can be effective to start with a core set of objectives and constraints and then iteratively increase the functionality and complexity of the model. This works best for models where a subset of the model still yields reasonable results, and for models that have primary and secondary functionality or goals (e.g., optimizing core and non-core resources). Let’s look at some examples.
- At the aforementioned fractional airline, we first optimized day-of planning of the jets. The client executives were pleased with that solution and then requested optimizing crew assignment and scheduling. At that point we proposed optimizing crew repositioning, the process in which pilots and flight attendants are ferried via third-party commercial airlines to other airports. By solving subproblems and earning user acceptance, we were able to add complexity as we progressed.
- For a railcar owner/operator, we were optimizing railcar distribution. The business priority was to meet customer demand while minimizing empty miles and unused railcars. After we successfully modeled according to those main objectives, we added minimizing deviation from railroad allotment and other soft operating constraints.
- For a network television company’s advertising revenue optimization initiative, we first modeled according to a customer’s set budget, assigning optimal slots for commercials. Subsequently, we added complexity by enhancing the granularity of the decision making and introducing additional objectives, like minimizing the budget to meet a goal for outcomes.
- When we consider a client with a Vehicle Routing Problem (VRP), we often start with driver rules and minimizing empty miles. Later, we add customer windows, layovers, different fleets and other complexities.
Defining “Done”
Agile is really about being “done” and shipping: you conduct your sprint demo and go to production. Then you go to the next sprint. This can be a very big challenge for an optimizer, as it can be difficult to know when a model is good enough.
To define “done,” it’s critical to have a good relationship with the product owner, set expectations, and establish clear criteria for the users. Feasibility instead of optimality might come first, for example, without a time limit, with performance speed addressed in a later iteration. Make the users understand that Agile in optimization is fluid: the solution will get better and deliver increasingly more value, and it takes time to get there.
Iterative Refinement
In Princeton’s projects, we typically employ a team member—who was not the modeler—to write a solution validator. See this blog post. The validator observes the rules for the solution and looks for violations. We will take a client’s existing schedule or solution and run it through a validator to see what and how many rules are broken—rules that client executives may have said were sacrosanct. Moreover, if we reinstate those rules, we sometimes get infeasibility, which then calls for optimization.
Test harnesses and test datasets are essential—they are widely used in Machine Learning but can be overlooked in optimization development. There are commercial no-code tools for testing that you can consider. At Princeton, we have developed in-house testing tools. A test harness allows the team to easily test model iterations with a variety of data, records the results, and compares against previous iterations, so the team can quickly identify when a change has been introduced and ensure it was expected. A robust set of test data, ideally coupled with the test harness, allows the team to track how solutions evolve over time, and ensures that the changes seen in the solutions are reasonable.
Introduce new functionality slowly: when adding new objectives and constraints, test against the test harness and dataset with each addition to identify which objective or constraint is contributing to solution changes. If adding a constraint with a cost, set the cost to zero initially to ensure that the addition of the constraint itself does not impact the solution.
But Will the Model Even Work?
The elephant in the room—my biggest worry and probably most practitioners’ as well— is whether the model will ultimately work. Will it find the optimal solution in a reasonable amount of time? Agile is no different than Waterfall in this regard, although there are practitioners who are scared away from Agile because they mistakenly believe it will impair success. I will remind them that plenty of Waterfall-developed solutions fail.
Final Thoughts
Set yourself up for Agile success by doing your homework, understanding the business, creating a road map, and embracing user engagement, which Agile facilitates. There are a lot of meetings in Agile, but let’s acknowledge that there are also many sessions in Waterfall, where for weeks upfront you typically create use‑case and functional design documents that are read line by line, followed by UAT sessions and fixes.
In Agile, you are testing and releasing more often. There can be a lot of overhead around that. Agile retrospectives are important, in my experience, especially with a product owner unused to optimization. Don’t skip the retros because you want to partner with the product owner and improve the process.
To discuss Agile for optimization solution development with Patricia, email us to set up a call.
