I'm having memory issues with the Rails application on Heroku, working with ruby 2.3.3 and Rails 4.1.13. When I download the application on Heroku, it starts at about 430 MB, but it is pretty stable, but according to Heroku it still reached 512 MB (as you would expect). My main question is: how to identify and reduce memory usage during BOOT in my application? That is, how to reduce the amount of memory used at startup?
I looked at most gems, for example. crash, oink, memory profiler and stackprof, but what helped me the most was Scout's addition to Heroku in terms of finding memory bloats and hungry controllers, etc. Thanks to this, I was able to reduce the amount of application that grows in terms of memory, but not so much how much memory it uses at startup.
I know that gems are loaded at startup, and the more and more, the more stones, the more memory is used at startup. Running bundle exec derailed bundle:mem gives me the following:
TOP: 92.3008 MiB carrierwave-aws: 16.7461 MiB carrierwave/storage/aws: 16.6094 MiB aws-sdk-core/s3: 7.9414 MiB aws-sdk-resources: 2.5703 MiB aws-sdk-core: 2.5117 MiB jmespath: 1.8867 MiB jmespath/nodes: 1.1406 MiB jmespath/nodes/function: 0.3867 MiB seahorse/client/base: 0.9414 MiB seahorse/client/plugins/net_http: 0.5273 MiB aws-sdk-core/xml/parser/engines/rexml: 0.8633 MiB rexml/document: 0.8516 MiB rexml/element: 0.5508 MiB rails/all: 13.6992 MiB rails: 8.9453 MiB (Also required by: active_record/railtie, active_model/railtie, and 6 others) rails/application: 7.2188 MiB rails/engine: 6.8672 MiB (Also required by: coffee/rails/engine) rails/railtie: 6.5898 MiB (Also required by: sprockets/railtie, jbuilder/railtie) rails/configuration: 6.3398 MiB (Also required by: rails/railtie/configuration) active_support/core_ext/object: 6.0586 MiB active_support/core_ext/object/conversions: 4.7227 MiB active_support/core_ext/hash/conversions: 4.0039 MiB (Also required by: active_support/core_ext/hash, active_record/serializers/xml_serializer) active_support/time: 3.8008 MiB (Also required by: active_record/base) active_support/core_ext/time: 3.0938 MiB active_support/core_ext/time/calculations: 2.9922 MiB (Also required by: active_support/core_ext/numeric/time, active_support/core_ext/string/conversions) active_support/core_ext/time/conversions: 2.2148 MiB (Also required by: active_support/core_ext/time, active_support/core_ext/date_time/conversions) active_support/values/time_zone: 2.1602 MiB (Also required by: active_support/time_with_zone, active_support/core_ext/date_time/conversions) tzinfo: 1.8984 MiB active_support/core_ext/array/conversions: 0.6211 MiB (Also required by: active_support/duration, action_dispatch/http/request, and 2 others) active_support/xml_mini: 0.5508 MiB (Also required by: active_support/core_ext/hash/conversions) active_support/core_ext/object/json: 1.0352 MiB (Also required by: active_support/json/encoding) json: 0.7227 MiB (Also required by: active_support/json/decoding, pg/text_encoder, and 19 others) json/common: 0.3906 MiB (Also required by: json/ext) active_support: 0.7109 MiB (Also required by: active_support/time, active_support/railtie, and 5 others) active_support/dependencies/autoload: 0.332 MiB (Also required by: rails, active_support/rails) active_support/inflector/methods: 0.332 MiB (Also required by: active_support/core_ext/string/inflections, active_support/core_ext/time/conversions, and 5 others) active_support/inflections: 0.3164 MiB (Also required by: active_support/inflector) active_support/logger: 0.3281 MiB action_dispatch/railtie: 0.668 MiB (Also required by: action_controller/railtie) action_dispatch: 0.6289 MiB (Also required by: action_controller) active_support/rails: 0.4414 MiB (Also required by: active_record, active_model, and 4 others) active_support/deprecation: 0.418 MiB (Also required by: sass/rails/helpers, formtastic/deprecation) active_record/railtie: 2.8555 MiB active_record: 1.9766 MiB (Also required by: friendly_id) active_record/connection_adapters/abstract_adapter: 0.8516 MiB arel: 0.8086 MiB (Also required by: active_record/base) arel/visitors: 0.3438 MiB action_controller/railtie: 0.875 MiB (Also required by: rails/all, sprockets/railtie) action_controller: 0.6797 MiB (Also required by: heroku_rails_deflate/serve_zipped_assets) action_controller/metal/live: 0.332 MiB sprockets/railtie: 1.8516 MiB (Also required by: sass/rails/railtie) sprockets/rails/helper: 0.918 MiB action_view/helpers: 0.8945 MiB (Also required by: action_view/base) action_view/helpers/form_helper: 0.3828 MiB (Also required by: action_view/helpers/form_options_helper) sprockets: 0.9102 MiB (Also required by: sprockets/rails/helper) sprockets/directive_processor: 0.5586 MiB yaml: 0.5078 MiB (Also required by: active_support/ordered_hash, hirb, and 7 others) psych: 0.5078 MiB rails3-jquery-autocomplete: 9.9102 MiB action_controller/base: 8.4922 MiB action_controller/metal: 1.8281 MiB abstract_controller/base: 1.0898 MiB erubis: 0.8477 MiB (Also required by: action_view/template/handlers/erb, heroics) erubis/engine: 0.4063 MiB (Also required by: erubis/engine/eruby) action_dispatch/middleware/stack: 0.6367 MiB active_support/dependencies: 0.5703 MiB (Also required by: abstract_controller/helpers, active_record/base) abstract_controller/rendering: 1.5625 MiB action_view/view_paths: 1.4531 MiB (Also required by: action_view/rendering) action_view/base: 1.3477 MiB action_view/lookup_context: 0.5156 MiB action_view/template: 0.3789 MiB (Also required by: action_view/template/resolver) action_controller/metal/redirecting: 1.0586 MiB action_controller/metal/rack_delegation: 0.9297 MiB action_dispatch/http/request: 0.8555 MiB action_controller/metal/request_forgery_protection: 0.5859 MiB action_controller/metal/params_wrapper: 0.5 MiB action_dispatch/http/mime_type: 0.375 MiB (Also required by: abstract_controller/collector, jbuilder/jbuilder_template) action_controller/metal/url_for: 0.3008 MiB rails3-jquery-autocomplete/formtastic: 1.2578 MiB formtastic/inputs/base: 0.8125 MiB fog: 8.6523 MiB fog/vcloud_director: 1.0156 MiB fog/vcloud_director/compute: 0.9961 MiB fog/vcloud_director/query: 0.4453 MiB pp: 0.418 MiB (Also required by: scout_apm) fog/openstack: 0.8398 MiB fog/cloudstack: 0.5859 MiB fog/cloudstack/compute: 0.5859 MiB fog/internet_archive: 0.543 MiB fog/internet_archive/storage: 0.5352 MiB fog/internet_archive/core: 0.3125 MiB fog/rackspace: 0.3867 MiB fog/cloudsigma: 0.3594 MiB fog/cloudsigma/compute: 0.3047 MiB fog/vcloud: 0.3203 MiB fog/vcloud/compute: 0.3125 MiB fog/aws: 0.3125 MiB twitter: 8.3711 MiB addressable/uri: 4.2422 MiB (Also required by: twitter/base, twitter/rest/request, and 8 others) addressable/idna: 2.7891 MiB addressable/idna/pure: 2.7461 MiB public_suffix: 0.3242 MiB twitter/streaming/client: 1.7266 MiB twitter/streaming/response: 1.4453 MiB http: 1.418 MiB http/response: 1.2109 MiB http/cookie_jar: 0.9219 MiB http/cookie: 0.9102 MiB domain_name: 0.8203 MiB domain_name/etld_data: 0.7813 MiB twitter/configuration: 0.8477 MiB (Also required by: twitter/rest/help) twitter/base: 0.6016 MiB (Also required by: twitter/entity, twitter/identity, and 12 others) twitter/null_object: 0.4688 MiB (Also required by: twitter/trend_results) naught: 0.4063 MiB twitter/cursor: 0.8164 MiB (Also required by: twitter/rest/utils, twitter/rest/friends_and_followers, and 2 others) twitter/rest/request: 0.7852 MiB (Also required by: twitter/rest/utils, twitter/rest/friends_and_followers, and 8 others) faraday: 0.6406 MiB (Also required by: twitter/rest/client, twitter/rest/response/parse_json, and 6 others) twitter/rest/client: 0.418 MiB twitter/rest/api: 0.3828 MiB asset_sync: 4.9609 MiB asset_sync/multi_mime: 4.5039 MiB mime/types: 4.4883 MiB (Also required by: mime/types/columnar) mime/types/registry: 4.1914 MiB asset_sync/storage: 0.4375 MiB fog/core: 0.4063 MiB (Also required by: fog, fog/xml, and 48 others) platform-api: 4.5273 MiB platform-api/client: 2.8359 MiB newrelic_rpm: 3.875 MiB new_relic/control: 3.875 MiB new_relic/agent: 3.4883 MiB new_relic/agent/agent: 1.7422 MiB new_relic/agent/configuration/manager: 0.4961 MiB (Also required by: new_relic/agent/configuration) new_relic/agent/configuration/default_source: 0.3633 MiB new_relic/agent/transaction_sampler: 0.3516 MiB carrierwave: 2.8281 MiB (Also required by: carrierwave-aws) carrierwave/uploader: 1.7656 MiB friendly_id: 2.5273 MiB friendly_id/object_utils: 2.4453 MiB active_record/base: 2.4258 MiB active_record/querying: 0.3906 MiB compass: 2.1328 MiB (Also required by: compass-rails) compass/sass_extensions: 1.2461 MiB compass/sass_extensions/sprites: 0.6875 MiB compass/sass_extensions/sprites/engines: 0.4609 MiB compass/sass_extensions/sprites/engines/chunky_png_engine: 0.4492 MiB chunky_png: 0.4492 MiB compass/sass_extensions/functions: 0.5078 MiB compass/configuration: 0.6172 MiB compass/configuration/data: 0.6133 MiB pg: 2.1055 MiB pg_ext: 1.9102 MiB feedjira: 2.0781 MiB loofah: 0.625 MiB sax-machine: 0.4414 MiB sass-rails: 1.8047 MiB sass/rails: 1.8008 MiB sass/rails/helpers: 1.7578 MiB sprockets/sass_functions: 1.7383 MiB sass: 1.7383 MiB (Also required by: sass/rails/importer, sprockets/sass_importer) sass/engine: 1.4492 MiB sass/script: 0.5586 MiB (Also required by: sass/script/css_parser) sass/scss: 0.3281 MiB sass/scss/parser: 0.3047 MiB scout_apm: 1.418 MiB webrick: 0.5664 MiB jbuilder: 1.1016 MiB (Also required by: jbuilder/jbuilder_template) active_support/cache: 0.3906 MiB rmagick: 1.0234 MiB rmagick_internal.rb: 1.0195 MiB RMagick2.so: 0.6875 MiB lazyload-rails: 0.8555 MiB nokogiri: 0.8555 MiB (Also required by: TOP, sax-machine/handlers/sax_nokogiri_handler, and 5 others) sitemap_generator: 0.5195 MiB dalli: 0.4648 MiB unicorn: 0.457 MiB hirb: 0.3984 MiB
I assume that 92 MB is pretty much in terms of gems, but I use them all in the application. Carrierwave is the biggest, but very important in how I cache images on AWS (as well as Fog) and so on. This amounts to approximately 1/4 of the memory used at boot.
In my application controller, I have a before filter where I load some variables that are used on the website (in global headers, sidebars, etc.) as follows:
@search = TextSearch.new @administrator_logged_in = current_administrator @nbr_of_stores_total = Store.active.size
I would suggest that this will be loaded into the boot memory (and in general), right?
Where else and how, should I look for a download at startup? In short:
Where in my rails application should I bother with boot memory (e.g. middleware, initializers, etc.)?
What tools (for example, a failed team) can be used to search / highlight hungry challenges / actions?
I'm having trouble finding the direct cause and effect of my troubleshooting steps. For example, finding the answer to the question "if I delete this code, how much memory will I save?". How can I measure / compare any actions that I perform in my application (rather than loading and checking overall usage on a chart)?