The Impact of android:debuggable on Android Application Startup Time

For several weeks members of my team and I have been trying to reconcile discrepancies in performance data for our application’s startup when measured locally and when measured on our CI platform. We weren’t really sure what was causing the discrepancies and resorted to brainstorming about the possible causes.

One of our most promising hypothesis was a theory that there was more of a difference between measurement techniques than we imagined. While we weren’t sure how the application’s startup time was being measured on the CI platform, we knew how it was being measured locally — using the Android Profiler.

We were testing the same build variant (a “release” build) locally and on the CI platform which meant that we were executing the same byte code on both. What other variables could account for the difference in measurements? It occurred to us that the CI platform might be taking its measurements with some tool other than the Android Profiler. One of the prerequisites of the Android Profiler is that the application under profile be built with the android:debuggable flag set to true.

Our team operated under the assumption that the value of this flag simply controlled an Android OS permission that dictated whether a debugger/profiler could attach to the process at runtime. In other words, we believed that the value of this flag should not change the way the Android OS’s runtime treated the application. However, this was the only potential difference we could imagine between the local and CI platform test environment, so we decided to investigate.

Turns out this was a good idea.

To properly isolate whether the debuggable flag is the cause of the change in startup performance, we had to figure out how to make two different versions of the application — both with the same byte code but one debuggable and the other not. In other words, we needed to generate two applications with the same bytecode but different values of the debuggable flag so that we could definitively attribute any differences in application start up time to the runtime. Ideally, then, we would toggle the flag on two copies of the same compiled version of the application (an apk).

The android:debuggable flag exists in an apk’s AndroidManifest.xml file. In older versions of Android, this file was, as the extension suggests, plain text xml. However, in modern versions of Android, the AndroidManifest.xml file is stored in a binary format in the apk.

There is not much publicly available documentation (that we could find quickly) about this file format. Several tools are available for inspecting its contents (e.g., aapt from the Android SDK) and we found some tools that promised to help modify the contents of an existing apk (e.g., apktool) but couldn’t get them to work properly.

That meant we were going to have to modify the binary-formatted xml file by hand! Fortunately, there is a good, open-source, standalone tool, axmldec, whose source code implicitly contains enough information about the file format to give us an idea of the binary format’s structure.

The android:debuggable flag is an attribute for an element. In the case of the android:debuggable flag, that element is the application itself. According to the source code, each element attribute is stored on disk in 20-byte structure:

ns
name
raw_value
value

4 bytes
4 bytes
4 bytes
8 bytes

Taking a cue from Java class file encoding techniques, this binary format stores every string in a table and the use of a string is stored simply as a pointer to an entry in that table. Because the namespace (ns) and name (name) of the attribute are strings, their values in this structure are 4-byte pointers into the string table. In this case, those strings are “android” and “debuggable”. The raw_value and value fields are more interesting. If the raw_value field is not 0xffffffff, then it represents a pointer to a string in the string table. In other words, the value of that attribute is some string and the value is meaningless. On the other hand, if the raw_value field is 0xffffffff, then the type of the element is some primitive whose value can be represented in 8 bytes.

A separate format defines the structure of the data in the raw_value field:

size
res0
data_type
data

2 bytes
1 byte
1 byte
4 bytes

The data and data_type type fields are the most useful. Again, according to the source code, a boolean (the type which we suppose the android:debuggable attribute to be) has a value of 0x12. For a boolean, “false” is 0x0000 in the data field and “true” is anything else.

With that understanding, we compiled a version of the application with the debuggable flag set to true and unzipped the resulting apk to get access to AndroidManifest.xml. Then, we used a modified version of axmldec to find the offset of the android:debuggable attribute in the application element. Using our newfound knowledge of the format of the elements, we used vi and xxd to manually change the attribute’s value from “true” to “false”. We used that modified AndroidManifest.xml file to create a non-debuggable version of the apk by zipping the contents of the directory generated when we unzipped the original apk. Finally, we signed the apk with our key and verified that our change to the debuggable flag worked ($ aapt d badging <path to apk> | grep -i debug).

Breath.

After all this work we learned …? Nothing. Yet.

We did, however, have a basis for a/b testing: Two applications, identical except for their debuggable flags.

We had to answer one final question before starting to test: How to fairly measure the application startup time? Remember, because only one of the two variants is marked as debuggable, we cannot use the Android Profiler. We found our answer in the am_ flags.

The Android Activity Monitor emits am_* flags when certain meaningful events occur. You can see a list of all the am_* flags here. We decided to rely on the am_proc_start flag which contains information about the time an application takes to start.

We didn’t need to build a raw data set where n was, say, 1000. We just needed something “rough and ready”. Our test tool ran each version of the application 5 times and recorded the am_proc_start times for each run. We recorded and analyzed the values.

You can see the raw data here.

Well, wow! The results indicate that our assumption was almost entirely wrong! Simply changing the value of the android:debuggable flag has a significant impact on the time it takes an application to start.

For us, the implications of this discovery are enormous.

  1. The measurements that we are taking of the time it takes to start our application on the CI platform and in the profiler cannot be compared directly with the reported startup times of comparable applications (unless, of course, they are also measuring versions of their application with the android:debuggable flag set).
  2. A corollary to (1) is that we can use timing results from Android Profiler and the CI platform to monitor performance changes over time. The caveat is that we have to be vigilant against the possibility that our optimizations are only improving the performance of startup components that the runtime executes when the debuggable flag is set. While these optimizations will not negatively affect startup performance under routine conditions, spending time on them would be a waste of engineering resources.
  3. We must, now, consider whether the value of the android:debuggable attribute changes other aspects of runtime performance. Although we do not believe that it does, these results show that we can no longer be certain of that.

As a result of this investigation, we are going to spend time researching where the runtime checks the debuggable flag and how it changes its behavior based on that value. We have several theories.

If you have any information leading to the arrest of the party responsible for the lost performance, please contact us in the comments!

Thank you to my fantastic teammate Anny who provided valuable feedback on a draft of this post. If you think that the post is well written and informative, it’s because of her. If you think that the post is confusing and boring, I take full responsibility.

Correcting ProGuard Configuration To Improve the Startup Time of Firefox’s New Mobile Browser

As a member of the performance engineering group at Mozilla, I am one of the people helping Firefox Preview’s development team with optimizing its startup time. Over several weeks, we have implemented many “easy” optimizations that significantly improved application startup time. Easy is in quotes to emphasize the irony — none of those startup performance optimizations were truly easy. The entire Firefox Preview team contributed to that effort and we were able to improve application startup by, primarily but not exclusively, deferring or parallelizing time-consuming startup tasks. More blog posts to come detailing the team’s earlier startup optimization achievements; the scope of this entry is more limited.

After shaking free all the low-hanging fruit from the startup speed tree, we wanted to step back and look at application startup time from a wider perspective. I began to wonder if there was a way that we could optimize the application’s on-disk representation to make it easier for the Android OS to load it into memory. I invited Jonathan Almeida, a talented, smart colleague into the investigation and the first thing we noticed was that on a Pixel 2 it took the OS more than three quarters of a second to load the application before even starting the Activity lifecycle.

Using the Android Profiler, we found that almost the entirety of application loading was being spent reading and parsing the application’s dex files. dex files contain the application’s byte code in Dalvik Executable format. Using Android Studio’s tools for analyzing an apk, the compressed file that contains all the application resources (including code, images, layouts, etc), Jonathan and I were able to inspect the size and composition of Firefox Preview’s dex files.

Figure 1
Figure 1

Figure 1 shows Studio’s Apk Analyzer at work. We saw that this version of Firefox Preview’s code requires three dex files, making it a so-called multi-dex application. Upon discovering that Firefox Preview’s code did not fit in a single dex file and because Jonathan knew that multi-dex files have implications for runtime performance, we hypothesized that they also negatively affect startup performance.

There are several reasons an application may end up with multiple dex files. Size is one of them. Firefox Preview has a robust set of dependencies and we weren’t sure how well we minimized and tracked the library dependencies listed in our Gradle build files. If there were dependencies listed that we did not use, removing those would reduce the size of Firefox Preview’s code and, Jonathan and I hoped, shrink the size of the bytecode to the point where it could fit in a single dex file.

Following best practices of Android development, we already used ProGuard within our build process to strip out unused methods and functions. The use of ProGuard means that a manual accounting of dependencies like this should have been unnecessary. Jonathan and I decided to do the manual inspection for excess code regardless, mostly out of curiosity. We found just a single dependent library that was listed in the Gradle build file but not actually used: Glide.

Glide is a library that ‘offers an easy to use API, a performant and extensible resource decoding pipeline and automatic resource pooling’ for ‘image loading … focused on smooth scrolling’.

The next step was to see if that code made it into Firefox Preview’s dex files. Again, because we were already using ProGuard, that code should not have been present. But, lo and behold, it was.

Figure 2

Figure 2 shows that the code for Glide did end up in Firefox Preview’s dex file. Relative to the overall size of Firefox Preview’s three dex files, removing these unused classes and methods was not going to have a major impact on their size.

Critically, though, it did indicate that something about our use of ProGuard was amiss.

There are a bewildering number of options available for configuring ProGuard. By complete accident, we noticed the -whyareyoukeeping option and configured ProGuard to tell us why it thought that it could not remove Glide from the output dex file despite the fact that Firefox Preview never instantiated any of its classes, used any of its static methods, etc.

Figure 3

Figure 3 shows ProGuard thought it needed to keep Glide’s code not because it was used but rather because it was obligated to follow a rule that told it to keep public and static methods from all classes. Those public methods, in turn, introduced a cascading set of dependencies on most of the library’s methods and classes.

The presence of such a catch-all rule was odd — it wasn’t anywhere in Firefox Preview’s codebase! Perhaps it was in the baseline rules included in the Android SDK? Nope, not there. We widened our search into the source code for the libraries that Firefox Preview uses. We didn’t get far before we found the rule in the ProGuard configuration for GeckoView (GV).

Neither Jonathan nor I is an expert at ProGuard, so it was a riddle how a ProGuard rule in a dependent library could “escape” and “infect” higher-level packages. We started to search through ProGuard information on the Internet and stumbled upon this “helpful” tidbit:

For library modules and for libraries distributed as AARs, the maintainer of the library can specify rules that will be supplied with the AAR and automatically exposed to the library consumer’s build system by adding this snippet to the module’s build.gradle file: … The rules that you put in the consumer-proguard.txt file will be appended to the main ProGuard configuration and used during the full application build.

Troubleshooting ProGuard issues on Android

Well, oops. GeckoView’s conservative ProGuard configuration seemed to be effectively negating Firefox Preview’s ProGuard code-shrinking optimizations. We were cautiously optimistic that fixing this issue would result in not only the removal of Glide’s methods and classes from Firefox Preview but an overall smaller amount of byte code and, perhaps, an application whose code fit entirely within a single dex file.

It was time to do some work. Jonathan and I removed that directive from GeckoView’s ProGuard configuration, rebuilt Firefox Preview and went back to our friend, the Android Studio Apk Analyzer.

Figure 4

Wow. From three dex files totaling almost 5.5MB to a single dex file totaling 3.4MB all with a single change. Major progress! But, there are still two outstanding questions.

First, why was Glide still referenced in the new, smaller dex file? Although there are fewer methods kept, it was puzzling to see it there at all. Remember, Firefox Preview never uses it! To answer this question, we went back to the trusty -whyareyoukeeping option in ProGuard. Again, we found that ProGuard kept those methods because of a configuration rule.

Figure 5

Funny, though, this rule existed nowhere in any of Mozilla’s code. greping through the Firefox Preview source code and the codebases of the various Mozilla components used by Firefox Preview confirmed this. The only remaining option was that a dependency used by Firefox Preview was doing the same thing that GeckoView was doing: leaking a static ProGuard configuration to its users that was overriding ProGuard’s willingness to strip Glide entirely from the output dex file. We guessed that it was more than likely in Glide itself.

Jonathan and I found that very ProGuard configuration directive in the Glide source code. The sad implication of this finding is that we could not rely on ProGuard to automatically remove Glide entirely and we would have to manually update Firefox Preview’s dependency configuration. Following the appropriate two-line change to Firefox Preview’s build.gradle file, we rebuilt Firefox Preview’s apk to test our work.

Figure 6

Whew. Glide is finally exorcised from the dex file!

Second, and more importantly: Did going from a multi-dex to a single-dex application have an impact on startup performance? The Android profiler could help Jonathan and I definitively answer this question. Figures 7 and 8 show the relevant portion of the profile of Firefox Preview startup before and after the ProGuard configuration change:

Figure 7
Figure 8

Profiling proved our initial hypothesis: multi-dex Android applications negatively effect startup performance and runtime performance.

The path Jonathan and I followed to achieve a decrease in the time that the Android OS took to load Firefox Preview led us on an Alice-In-Wonderland-like trip where we found ourselves profiling application runtime, analyzing dex files and apks and learning more about ProGuard than we ever wanted to know. The savings we gained was far more than we could have imagined when we started and more than justified the time we spent traveling this long and winding road.

Update: This post has been edited to remove references to Mozilla-internal names and to explicitly reference my previously unnamed, mystery collaborator.

Update 2: Removed the first paragraph because it contained an implicit promise that I did not intend.