Support perf-cost experiment for local backend

[mcopik]

The local backend that deploys functions in Docker containers on the user machine has been developed for testing purposes. However, our users want to use this backend for invocations and experiments, and we should also support this. Currently, the local backend does not have the full functionality of a FaaS platform.

• Implement missing parts of the faas.System interface.
• Add HTTP triggers.
• Support cold invocations with container restart.
• Add limits on the number of parallel containers to avoid too much CPU and memory overhead.

[Kaushik-Iyer]

I would like to try to work on this issue? Can you please guide me on which part of the codebase to focus on?

[mcopik]

@Kaushik-Iyer Thank you for your interest in the project!

You can find the perf-cost experiment here. The experiment's goal is to invoke a function N times, in batches, with cold and warm starts. You can find an example of using it with clouds in documentation.

As you can see, it takes an instance of FaaSSystem (line 42) and uses it later to create functions, deploy them, and invoke them. Examples of FaaSSystem are provided in sebs.aws, sebs.azure, sebs.gcp, and sebs.openwhisk. We have a module sebs.local but its implementation is not as complete as the other systems.

You can start first by running the local function deployment (see docs) - this will create Docker containers running a function's instance. Test it and see how you can invoke functions. Then, try to run the experiment with local deployment - keep the number of functions in a batch low to prevent overloading your system with too many containers. You should be able to experimentally verify which features are missing.

The HTTP trigger for local deployment is actually already implemented - see sebs.local.function.HTTPTrigger.

[Rajiv2605]

@Kaushik-Iyer are you working on this issue? Can I take it up?

[Kaushik-Iyer]

You can work on it

[mcopik]

@Rajiv2605 Please let me know if you have any questions - happy to help.

[Rajiv2605]

@mcopik thanks! I will follow what you have suggested in the previous comment. Please let me know if there is any additional information I need to know.

[mcopik]

Sounds good - please ask if you find anything unclear or if the documentation is missing/incomplete.

[Rajiv2605]

@mcopik I am able to run benchmark in local. But perf-cost is throwing errors.

[veenaamb]

Can you post your error? @Rajiv2605

[Rajiv2605]

rajiv@DESKTOP-B3BCG09:~/serverless-benchmarks$ sudo ./sebs.py experiment invoke perf-cost --config config/example.json --deployment local

ERROR:root:'Local' object has no attribute '_measure_interval'
Traceback (most recent call last):
  File "./sebs.py", line 30, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "./sebs.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "./sebs.py", line 97, in wrapper
    return func(*args, **kwargs)
  File "./sebs.py", line 462, in experiment_invoke
    experiment.prepare(sebs_client, deployment_client)
  File "/home/rajiv/serverless-benchmarks/sebs/experiments/perf_cost.py", line 49, in prepare
    self._function = deployment_client.get_function(self._benchmark)
  File "/home/rajiv/serverless-benchmarks/sebs/faas/system.py", line 174, in get_function
    function = self.create_function(code_package, func_name)
  File "/home/rajiv/serverless-benchmarks/sebs/local/local.py", line 194, in create_function
    if self.measurements_enabled and self._memory_measurement_path is not None:
  File "/home/rajiv/serverless-benchmarks/sebs/local/local.py", line 54, in measurements_enabled
    return self._measure_interval > -1
AttributeError: 'Local' object has no attribute '_measure_interval'

[Rajiv2605]

@

Can you post your error? @Rajiv2605
Please check the error above. Did I miss any step?

[Rajiv2605]

rajiv@DESKTOP-B3BCG09:~/serverless-benchmarks$ sudo ./sebs.py experiment invoke perf-cost --config config/example.json --deployment local --verbose

[sudo] password for rajiv:
19:06:09,721 INFO SeBS-0912: Created experiment output at /home/rajiv/serverless-benchmarks
19:06:09,724 INFO Benchmark-d250: Using cached benchmark 110.dynamic-html at /home/rajiv/serverless-benchmarks/cache/110.dynamic-html/local/python/3.7/code
19:06:09,724 INFO Local-415d: Creating new function! Reason: function 110.dynamic-html-python-3.7 not found in cache.
ERROR:root:'Local' object has no attribute '_measure_interval'
Traceback (most recent call last):
  File "./sebs.py", line 30, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "./sebs.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "./sebs.py", line 97, in wrapper
    return func(*args, **kwargs)
  File "./sebs.py", line 462, in experiment_invoke
    experiment.prepare(sebs_client, deployment_client)
  File "/home/rajiv/serverless-benchmarks/sebs/experiments/perf_cost.py", line 49, in prepare
    self._function = deployment_client.get_function(self._benchmark)
  File "/home/rajiv/serverless-benchmarks/sebs/faas/system.py", line 174, in get_function
    function = self.create_function(code_package, func_name)
  File "/home/rajiv/serverless-benchmarks/sebs/local/local.py", line 194, in create_function
    if self.measurements_enabled and self._memory_measurement_path is not None:
  File "/home/rajiv/serverless-benchmarks/sebs/local/local.py", line 54, in measurements_enabled
    return self._measure_interval > -1
AttributeError: 'Local' object has no attribute '_measure_interval'