Skip to content

[microTVM] Refactor RVM scripts and fix DNS network issue #11741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 16, 2022
Merged

[microTVM] Refactor RVM scripts and fix DNS network issue #11741

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2e032c4
refactor scripts
mehrdadh Jun 15, 2022
0b7df04
address comments
mehrdadh Jun 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 6 additions & 29 deletions apps/microtvm/reference-vm/arduino/base-box/base_box_setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,42 +26,19 @@ if [ -e "$HOME/skip_zeroing_disk" ]; then
skip_zeroing_disk=1
fi

sudo apt update
sudo apt install -y build-essential
sudo apt-get --purge remove modemmanager # required to access serial ports.
# Install common configs
~/base_box_setup_common.sh
rm -f ~/base_box_setup_common.sh

sudo apt install -y --no-install-recommends git \
cmake cmake-data \
ninja-build gperf ccache dfu-util device-tree-compiler wget \
python3-dev python3-pip python3-setuptools python3-tk python3-wheel xz-utils file \
make gcc gcc-multilib g++-multilib libsdl2-dev

OLD_HOSTNAME=$(hostname)
sudo hostnamectl set-hostname microtvm
sudo sed -i.bak "s/${OLD_HOSTNAME}/microtvm.localdomain/g" /etc/hosts

# Poetry deps
sudo apt install -y python3-venv

# TVM deps
sudo apt install -y llvm

# ONNX deps
sudo apt install -y protobuf-compiler libprotoc-dev
# Poetry
sed -i "/^# If not running interactively,/ i source \$HOME/.poetry/env" ~/.bashrc
Copy link
Member

@guberti guberti Jun 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment explaining this line? Why are we looking for "If not running interactively"?

Copy link
Member Author

@mehrdadh mehrdadh Jun 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is just added to top of the .bashrc file. Also I only moved this part from other scripts

Copy link
Member

@guberti guberti Jun 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just add the line to the top of the file then, instead of relying on the exact wording of the comment? Say,

sed -i "1i source \$HOME/.poetry/env" ~/.bashrc

May be out of scope for this PR, though.

sed -i "/^# If not running interactively,/ i\\ " ~/.bashrc

# TODO do we need this?
echo 'export PATH=$HOME/vagrant/bin:"$PATH"' >> ~/.profile
source ~/.profile
echo PATH=$PATH

# Poetry
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3
sed -i "/^# If not running interactively,/ i source \$HOME/.poetry/env" ~/.bashrc
sed -i "/^# If not running interactively,/ i\\ " ~/.bashrc

# Python 3.7
sudo apt install -y python3.7

# Clean box for packaging as a base box
sudo apt-get clean
if [ $skip_zeroing_disk -eq 0 ]; then
Expand Down
9 changes: 8 additions & 1 deletion apps/microtvm/reference-vm/base-box-tool.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,15 @@
# Extra scripts required to execute on provisioning
# in [platform]/base-box/base_box_provision.sh
EXTRA_SCRIPTS = {
"arduino": (),
"arduino": (
"apps/microtvm/reference-vm/base_box_setup_common.sh",
"docker/install/ubuntu_install_core.sh",
"docker/install/ubuntu_install_python.sh",
),
"zephyr": (
"apps/microtvm/reference-vm/base_box_setup_common.sh",
"docker/install/ubuntu_install_core.sh",
"docker/install/ubuntu_install_python.sh",
"docker/install/ubuntu_init_zephyr_project.sh",
"docker/install/ubuntu_install_zephyr_sdk.sh",
"docker/install/ubuntu_install_cmsis.sh",
Expand Down
66 changes: 66 additions & 0 deletions apps/microtvm/reference-vm/base_box_setup_common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
#!/bin/bash -e
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -e
set -x

# Fix network DNS issue
sudo sed -i 's/DNSSEC=yes/DNSSEC=no/' /etc/systemd/resolved.conf
sudo systemctl restart systemd-resolved

sudo apt update
sudo apt install -y build-essential
sudo apt-get --purge remove modemmanager # required to access serial ports.

# Core
sudo ~/ubuntu_install_core.sh
rm -f ~/ubuntu_install_core.sh

sudo apt install -y --no-install-recommends git \
gperf ccache dfu-util device-tree-compiler xz-utils file \
gcc gcc-multilib g++-multilib libsdl2-dev

# Cmake
wget --no-verbose https://apt.kitware.com/keys/kitware-archive-latest.asc
sudo apt-key add kitware-archive-latest.asc
sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main'
sudo apt update
sudo apt install -y --no-install-recommends \
cmake=3.22.2-0kitware1ubuntu18.04.1 cmake-data=3.22.2-0kitware1ubuntu18.04.1 \

# Python
sudo ~/ubuntu_install_python.sh
rm -f ~/ubuntu_install_python.sh
Copy link
Member

@guberti guberti Jun 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these scripts write protected? Do we need the -f flag either?


# Poetry deps
sudo apt install -y python3-venv

# TVM deps
# TODO(mehrdadh): replace with ubuntu_install_llvm.sh
sudo apt install -y llvm

# ONNX deps
sudo apt install -y protobuf-compiler libprotoc-dev

# Poetry
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3

# Host name
OLD_HOSTNAME=$(hostname)
sudo hostnamectl set-hostname microtvm
sudo sed -i.bak "s/${OLD_HOSTNAME}/microtvm.localdomain/g" /etc/hosts
47 changes: 7 additions & 40 deletions apps/microtvm/reference-vm/zephyr/base-box/base_box_setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,38 +26,14 @@ if [ -e "$HOME/skip_zeroing_disk" ]; then
skip_zeroing_disk=1
fi

sudo apt update
sudo apt install -y build-essential
sudo apt-get --purge remove modemmanager # required to access serial ports.
# Install common configs
~/base_box_setup_common.sh
rm -f ~/base_box_setup_common.sh

# Zephyr
wget --no-verbose https://apt.kitware.com/keys/kitware-archive-latest.asc
sudo apt-key add kitware-archive-latest.asc
sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main'
sudo apt update
# NOTE: latest cmake cannot be installed due to
# https://github.com/zephyrproject-rtos/zephyr/issues/30232
sudo apt install -y --no-install-recommends git \
cmake=3.22.2-0kitware1ubuntu18.04.1 cmake-data=3.22.2-0kitware1ubuntu18.04.1 \
ninja-build gperf ccache dfu-util device-tree-compiler wget \
python3-dev python3-pip python3-setuptools python3-tk python3-wheel xz-utils file \
make gcc gcc-multilib g++-multilib libsdl2-dev

# Avahi, so that ssh microtvm works.
# apt install -y avahi-daemon

OLD_HOSTNAME=$(hostname)
sudo hostnamectl set-hostname microtvm
sudo sed -i.bak "s/${OLD_HOSTNAME}/microtvm.localdomain/g" /etc/hosts

# Poetry deps
sudo apt install -y python3-venv

# TVM deps
sudo apt install -y llvm

# ONNX deps
sudo apt install -y protobuf-compiler libprotoc-dev
# Poetry
sed -i "/^# If not running interactively,/ i source \$HOME/.poetry/env" ~/.bashrc
sed -i "/^# If not running interactively,/ i export ZEPHYR_BASE=$HOME/zephyr/zephyr" ~/.bashrc
sed -i "/^# If not running interactively,/ i\\ " ~/.bashrc

# nrfjprog
NRF_COMMANDLINE_TOOLS_FILE=nRFCommandLineToolsLinuxamd64.tar.gz
Expand Down Expand Up @@ -95,15 +71,6 @@ sudo apt install -y python3.8-dev
sudo find ~/zephyr-sdk -name '*.rules' -exec cp {} /etc/udev/rules.d \;
sudo udevadm control --reload

# Poetry
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3
sed -i "/^# If not running interactively,/ i source \$HOME/.poetry/env" ~/.bashrc
sed -i "/^# If not running interactively,/ i export ZEPHYR_BASE=$HOME/zephyr/zephyr" ~/.bashrc
sed -i "/^# If not running interactively,/ i\\ " ~/.bashrc

# Python 3.7
sudo apt install -y python3.7

# Clean box for packaging as a base box
sudo apt-get clean
if [ $skip_zeroing_disk -eq 0 ]; then
Expand Down
3 changes: 1 addition & 2 deletions apps/microtvm/reference-vm/zephyr/base-box/base_box_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,4 @@ fi

board=$1

pytest tests/micro/zephyr/test_zephyr.py --zephyr-board=${board}
pytest tests/micro/zephyr/test_zephyr_aot.py --zephyr-board=${board}
pytest tests/micro/zephyr --zephyr-board=${board}
2 changes: 2 additions & 0 deletions docker/install/ubuntu_install_core.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

set -e
set -u
# Used for debugging RVM build
set -x
set -o pipefail

# install libraries for building c++ core on ubuntu
Expand Down
7 changes: 6 additions & 1 deletion docker/install/ubuntu_install_python.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

set -e
set -u
# Used for debugging RVM build
set -x
set -o pipefail

# install python and pip, don't modify this, modify install_python_package.sh
Expand All @@ -33,8 +35,11 @@ apt-get install -y python-pip python-dev python3.6 python3.6-dev

rm -f /usr/bin/python3 && ln -s /usr/bin/python3.6 /usr/bin/python3

# python 3.7
apt-get install -y python3.7

# Install pip
cd /tmp && wget -q https://bootstrap.pypa.io/get-pip.py && python3.6 get-pip.py
wget -q https://bootstrap.pypa.io/get-pip.py && python3.7 get-pip.py

# Pin pip and setuptools versions
pip3 install pip==19.3.1 setuptools==58.4.0