Config Updates Required for ActivitySim 1.3.5

[JoeJimFlood]

To get a head start on identifying what changes we would need to make to our configuration files for ActivitySim v1.3.5, we conducted a full run of ABM3 with the current state of the repo. There were a couple of model crashes that were likely consequences of the upgrade to Pandas 2.x. While these aren't issues that we need (or have the time to wait for) to be fixed, I did want to let the community know about them in case other ActivitySim users who want to use v1.3.5 need to make the same changes to their configuration files or extensions.

Deprecation of DataFrame.append()

One of the changes that Pandas made for version 2 was that the append method for the DataFrame object was deprecated. This method was called in our airport_returns extension that is used in our airport access models. The workaround for this is fairly straightforward, as df.append(df2) can be replaced by df = pd.concat([df, df2]). This change can be seen in this commit.

Stricter enforcement of categories when updating a categorical field

The trip modes that are chosen in our airport access models are actually the arrival modes that people use to access the airport. For example, different parking or ridehail locations are recorded as different modes and there are additional options not present in other models (such as hotel courtesy shuttles). Because these are different from the modes that we use for assignment, the write trip matrices preprocessor copies the trip_mode column to a column called arrival_mode. The values are then recoded to match the trip modes that are used in assignment. This involves adding categories to the trip_mode field, which worked with Pandas 1.x but not Pandas 2.x. To get around this, we created a new column called trip_mode_assign and had the trip tables used for assignment be defined based on those values. However, our internal reporting relies on the trip_mode column of the airport model output files having the correct categories, so a post-processing step was added to delete the trip_mode field (that information is already stored in arrival_mode and rename the trip_mode_assign field to trip_mode. These changes can be seen in this commit.

[bhargavasana]

Here is some more information regarding the stricter enforcement of categories point above.

We were trying to recode trip mode in our airport ground access model - https://github.com/SANDAG/ABM/blob/ABM3_develop/src/asim/configs/common_airport/write_trip_matrices_annotate_trips_preprocessor.csv#L34. However, that results in an error because we need to add new categories to the trip_mode column before the update occurs.

We would be interested in hearing about a better way to overcome this hurdle. Thanks!

[dhensle]

I think we need to update the logic in the assign_in_place function. I believe moving the handling of categoricals to before df.update(df2) is called and updating the logic a bit around this change should solve the error.

Tagging @i-am-sijia to see if she ran into this elsewhere during the categorical data development work and has a different solution in mind.

[i-am-sijia]

The crash in the airport model happens when the preprocessor tries to update an existing categorical column in df with new category values from df2 by calling df.update(df2). This is a scenario that none of the existing example ActivitySim model cover, and therefore it wasn't caught in the recent Pandas 2 PR.

Say df and df2 have the same columns but slightly different values; df uses categorical dtype, df2 may or may not use categorical dtype.

If we update/overwrite df with values from df2, by running df.update(df2):

• In Pandas 1.x, it does not error, but the final column in df becomes an object (string) type to include the new values from df2

  

• In Pandas 2.x, it raises a TypeError as shown in the SANDAG airport model

  

Thoughts on solution

When I implemented categoricals for ActivitySim with pandas 1.x, I encountered a similar but slightly different issue with pd.concat() - in appending two categoricals with different category values (e.g., the tours table before appending at-work tours has three categories for tour_category: mandatory, non-mandatory, and joint, while the at-work tour has a new category value: atwork), Pandas 1.x does not raise an error. Instead it silently converts the resulting column back to object (string) type to accommodate the new category value, which negates the memory saving of using categoricals.

The solution I implemented at the time was to first take the union of the two categoricals before appending. See [1][2]. I believe a similar approach can be applied to the current issue. I've implemented a solution in PR #948.

@JoeJimFlood will you please check if that resolves the crash in your airport model?