mcsema-lift crash (Segmentation fault)

[FredyR4zox]

Hi,

When using mcsema-lift, I get an segmentation fault.
Docker image: https://hub.docker.com/layers/trailofbits/mcsema/llvm1100-ubuntu20.04-amd64/images/sha256-50e3e04fdcce8e83ea801542dcf7fdd902306674028a829ab0dcaa0161a07a72?context=explore

File: test3.c

#include <stdio.h>

int get_sign(int x) {
  if (x == 0)
    return 0;

  if (x < 0)
    return -1;
  else
    return 1;
}

int main() { 
  int a;

  scanf("%d", &a);

  int res = get_sign(a);

  if(res > 0)
    printf("%d is positive\n", a);
  else if(res < 0)
    printf("%d is negative\n", a);
  else
    printf("%d is zero\n", a);

  return 0;
}

> clang --version                                                                                                                         
clang version 11.0.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
> clang -g -O0 -fno-stack-protector -no-pie -Wall test3.c -o test3
> '/home/fredyr4zox/Downloads/IDA Pro/IDA Pro 7.5/idat64.exe' -S"/home/fredyr4zox/.local/lib/python3.8/site-packages/mcsema_disass-3.1.3.8-py3.8.egg/mcsema_disass/ida7/get_cfg.py --output /home/fredyr4zox/Builds/Thesis/test3.cfg --log_file /home/fredyr4zox/Builds/Thesis/log --arch amd64 --os linux --entrypoint main" /home/fredyr4zox/Builds/Thesis/test3
> sudo docker run -it --ipc=host --entrypoint=/bin/bash -v /home/fredyr4zox/Downloads/IDA\ Pro/:/mcsema/ida -v "$(pwd)":/mcsema/local trailofbits/mcsema:llvm1100-ubuntu20.04-amd64
root@692d6bc95e1c:/# cd /mcsema/local/
root@692d6bc95e1c:/mcsema/local# mcsema-lift-11.0 --arch amd64 --os linux --cfg test3.cfg --output test3.bc
Segmentation fault (core dumped)
root@692d6bc95e1c:/mcsema/local# // I installed GDB and GEF for GDB
root@692d6bc95e1c:/mcsema/local# gdb mcsema-lift-11.0
.......................
(gdb) r --arch amd64 --os linux --cfg test3.cfg --output test3.bc
.......................
     0x54f0c1 <remill::CallersOf(llvm::Function*)+97> cmp    BYTE PTR [r15+0x10], 0x17
     0x54f0c6 <remill::CallersOf(llvm::Function*)+102> mov    ecx, 0x0
     0x54f0cb <remill::CallersOf(llvm::Function*)+107> cmova  rcx, r15
 →   0x54f0cf <remill::CallersOf(llvm::Function*)+111> movzx  edx, BYTE PTR [rcx+0x10]
     0x54f0d3 <remill::CallersOf(llvm::Function*)+115> cmp    dl, 0x1d
     0x54f0d6 <remill::CallersOf(llvm::Function*)+118> je     0x54f0b0 <remill::CallersOf(llvm::Function*)+80>
     0x54f0d8 <remill::CallersOf(llvm::Function*)+120> cmp    dl, 0x23
     0x54f0db <remill::CallersOf(llvm::Function*)+123> je     0x54f0b0 <remill::CallersOf(llvm::Function*)+80>
     0x54f0dd <remill::CallersOf(llvm::Function*)+125> cmp    dl, 0x50
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── threads ────
[#0] Id 1, Name: "mcsema-lift-11.", stopped 0x54f0cf in remill::compat::llvm::CallSite::CallSite (), reason: SIGSEGV
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── trace ────
[#0] 0x54f0cf → remill::compat::llvm::CallSite::CallSite(this=<optimized out>, user=0x360bec0)
[#1] 0x54f0cf → remill::CallersOf(func=<optimized out>)
[#2] 0x4793eb → mcsema::(anonymous namespace)::ReplaceUndefIntrinsic(function=0x27d63f8)
[#3] 0x4793eb → mcsema::(anonymous namespace)::RemoveUndefFuncCalls()
[#4] 0x4793eb → mcsema::CleanUpModule(cfg_module=0x327a270)
[#5] 0x4750f1 → mcsema::LiftCodeIntoModule(cfg_module=0x327a270)
[#6] 0x48a34c → main(argc=<optimized out>, argv=<optimized out>)

This is very weird since it is a simple program.

The zip file attached has all the files. The .c file, the compiled program, and the 3 files from the mcsema-disass comand.
mcsema-lift-issue.zip

[xiaoyong-z]

i have the same problem, did you fix it?

[FredyR4zox]

@xiaoyong-z Unfortunately no :/
How are you using it? With IDA in TUI mode, just like me?

[xiaoyong-z]

well, i use IDA the same way like you.
And just now i used llvm-9 to build another version of mcsema-lift(mcsema-lift-9), it works for me, i can use it to lift cfg to llvm bytecode.
I still have no idea why mcsema-lift-11.0 not works, but i think mcsema-lift-9 is worth a try.

[ekilmer]

Hello. Apologies for the late response and segfault on LLVM-11. LLVM-11 support for McSema is new and hasn't gone through any sort of meaningful testing on CI (our fault).

I'm trying to figure it out right now and I have some work-in-progress branches for remill (lifting-bits/remill#495) and mcsema (#739) to try and resolve this.

In the meantime, I think LLVM 10 and LLVM 9 versions will work as @xiaoyong-z says.

Thank you all for being patient.

[ekilmer]

I think this should be fixed now on master branch, and I can lift your example without failure. Could you please confirm?

$ mcsema-lift-11.0 --arch amd64 --os linux --cfg /Users/ekilmer/Downloads/mcsema-lift-issue/test3.cfg --output issue.bc
$ echo $?
0

[FredyR4zox]

Thank you for the response.

I cloned the updated master branch and built the docker image. (Before this step i deleted all my docker images).
Still, it is giving me the same error on the same line :/

[ekilmer]

Oof. Apologies again 😬 . Currently, the Docker images are no longer faithfully pulling the latest remill and anvill correctly. This is an issue with the recent changes made to remill and anvill since they no longer produce the old Docker images that the McSema Docker image relies on.

I will work on fixing the Docker image to build everything from source using the new vcpkg method later today.

If you'd like to attempt building remill, anvill, and mcsema from source on Linux then you can try following a similar workflow as what's in CI for vcpkg:

mcsema/.github/workflows/vcpkg_ci.yml

Lines 41 to 70 in 2ed00dc

	- name: Build and install remill
	shell: bash
	run: |
	# Change to master after testing
	git clone --depth=1 --branch master https://github.com/lifting-bits/remill.git
	cd remill
	./scripts/build.sh --llvm-version ${{ matrix.llvm }}
	cmake --build remill-build --target install -- -j "$(nproc)"
	- name: Build and install anvill
	shell: bash
	run: |
	# Change to master after testing
	git clone --depth=1 --branch master https://github.com/lifting-bits/anvill.git
	mkdir -p anvill/build && cd anvill/build
	cmake -DCMAKE_VERBOSE_MAKEFILE=ON ..
	cmake --build . --target install -- -j "$(nproc)"
	- name: Build mcsema
	shell: bash
	run: |
	mkdir build && cd build
	cmake -DCMAKE_VERBOSE_MAKEFILE=ON ..
	cmake --build . --target install -- -j "$(nproc)"
	- name: Smoke Test
	shell: bash
	run: |
	# --help return non-zero and fails...
	mcsema-lift-${{ matrix.llvm }}.0 --version
	mcsema-disass-2 --help
	mcsema-lift-${{ matrix.llvm }}.0 --arch amd64 --os linux --cfg ./tests/test_suite_generator/generated/prebuilt_cfg/amd64/linux/cfg/hello_world --output hello_world.bc
	mcsema-lift-${{ matrix.llvm }}.0 --arch x86 --os linux --cfg ./tests/test_suite_generator/generated/prebuilt_cfg/x86/linux/cfg/hello_world --output hello_world.bc

As for native dependencies required, this is how our CI environment is set up: https://github.com/trailofbits/cxx-common/blob/69eeb951886a69e45c9b9d5369768e1a6dff4a2a/docker/Dockerfile.ubuntu.vcpkg#L13-L24 with clang-10 as the compiler.

Otherwise, using the current Docker images for mcsema on LLVM versions 9 and 10 should still work.

[FredyR4zox]

Makes sense then :)

I tried with LLVM 9 and LLVM 10 versions from docker hub and it worked perfectly. Thank you.

But i have another problem. Which is when running the lifted code with KLEE.
I tried the Maze example in https://github.com/lifting-bits/mcsema/tree/master/examples/Maze and downloaded the CFG maze.amd64.cfg from the example folder (just to make sure the problem wasn't with my IDA Pro).
The mcsema-lift run successfully and it compiles back to a binary format. But when running with KLEE, the following error happens:

>>> sudo docker run -it --ipc=host --entrypoint=/bin/bash -v /home/fredyr4zox/Downloads/IDA\ Pro/:/mcsema/ida -v "$(pwd)":/mcsema/local trailofbits/mcsema:llvm900-ubuntu20.04-amd64
root@a0e223615859:/# cd /mcsema/local
root@a0e223615859:/mcsema/local# mcsema-lift-9.0 --arch amd64 --os linux --cfg maze.amd64.cfg --explicit_args --output maze_amd64.bc
root@a0e223615859:/mcsema/local# Ctrl+D
>>> sudo docker run --rm -ti -v "$(pwd)":/home/klee/thesis/ --ulimit='stack=-1:-1' klee/klee:2.2
klee@7be503443453:~$ cd thesis
klee@7be503443453:~/thesis$ klee \
    --simplify-sym-indices \
    --solver-backend=z3 \
    --solver-optimize-divides \
    --use-forked-solver \
    --use-independent-solver \
    --write-cov \
    --write-paths \
    --write-sym-paths \
    --write-test-info \
    --external-calls=all \
    --suppress-external-warnings \
    --posix-runtime \
    --libc=none \
    maze_amd64.bc --sym-stdin 28
KLEE: NOTE: Using POSIX model: /tmp/klee_build90stp_z3/runtime/lib/libkleeRuntimePOSIX64_Debug+Asserts.bca
KLEE: output directory is "/home/klee/thesis/klee-out-4"
KLEE: Using Z3 solver backend
KLEE: WARNING: undefined reference to function: _ITM_deregisterTMCloneTable
KLEE: WARNING: undefined reference to function: _ITM_registerTMCloneTable
KLEE: WARNING: undefined reference to function: __libc_start_main
KLEE: WARNING: undefined reference to function: printf
KLEE: WARNING: undefined reference to function: sleep
KLEE: WARNING: undefined reference to function: strlen
KLEE: WARNING: undefined reference to function: strncmp
KLEE: WARNING ONCE: calling __gmon_start__ with extra arguments.
KLEE: ERROR: (location information missing) return void when caller expected a result
KLEE: NOTE: now ignoring this error at this location

KLEE: done: total instructions = 171
KLEE: done: completed paths = 1
KLEE: done: generated tests = 1
klee@7bc2202adf46:~/thesis$ 

All of this was done using the version 9 of LLVM.

klee@7be503443453:~/thesis$ klee --version
KLEE 2.2 (https://klee.github.io)
  Build mode: RelWithDebInfo (Asserts: TRUE)
  Build revision: 5719d2803e93252e5d4613f43afc7db0d72332f1

LLVM (http://llvm.org/):
  LLVM version 9.0.1
  Optimized build with assertions.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: sandybridge

root@63f5cc717b2e:/mcsema/local# mcsema-lift-9.0 --version
mcsema-lift-9.0 version v3.0.23
Commit Hash: 2ed00dc29e85db08e099cd067db6a3a47549b915
Commit Date: 2021-03-02 16:05:45 -0500
Last commit by: Eric Kilmer [[email protected]]
Commit Subject: [Fix bad LoadInst compat for LLVM 11 (#739)]

Uncommitted changes were present during build.

Do you have any idea of what it might be?

[ekilmer]

Hmmm I'm not exactly sure, regarding the KLEE question. I think @pgoodman might have a better idea.

@FredyR4zox I have an open PR here #741 that tries to fix the LLVM 11 Docker image. Would you be able to test out that branch?

[FredyR4zox]

Thank you, I will test it until the end of tomorrow.
I can't use the docker to build it right? Because of the problems you listed above

[ekilmer]

You should be able to do something like

$ docker build -t mcsema .

in that branch and then I was able to run the following a zero return code

docker run --rm -t -i -v ~/Downloads/mcsema-lift-issue:/work mcsema --arch amd64 --os linux --cfg /work/test3.cfg --output /work/issue.bc

or shell into the Docker container similar to what you posted in the OP

[pgoodman]

Is the call to __gmon_start__ the issue? That is a function to initialize gprof profiling. We declare it somewhere in mcsema just so that its prototype is available. It is possible that our declaration is slightly wrong. It is usually a weakly-defined symbol.

[ekilmer]

@FredyR4zox now that #741 is merged, pulling the docker image should work now.

$ docker pull trailofbits/mcsema:llvm11-ubuntu20.04-amd64
$ docker run --rm -t -i -v ~/Downloads/mcsema-lift-issue:/work \
    trailofbits/mcsema:llvm11-ubuntu20.04-amd64 \
    --arch amd64 --os linux --cfg /work/test3.cfg --output /work/issue.bc

Note the lack of trailing 00 in the image name for llvm11

[FredyR4zox]

@ekilmer In the docker image trailofbits/mcsema:llvm11-ubuntu20.04-amd64, iIinstalled wine to be able to run IDA Pro with python from windows (I have IDA Pro for windows). When I ran IDA Pro in TUI mode to disassemble the binary, at the end it gave the following error: AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'. I search for the error and the solution was to upgrade the protobuf package. I upgrade it and it gave a warning that mcsema-disass had a dependency of the old version of protobuf.

root@ae099e3243a6:/mcsema/local# DISPLAY=:0.0 WINEPREFIX=~/.wine64 wine pip install protobuf
Requirement already satisfied: protobuf in z:\opt\trailofbits\lib\python3\site-p
ackages\protobuf-3.2.0-py3.8.egg (3.2.0)
Requirement already satisfied: setuptools in c:\users\root\local settings\applic
ation data\programs\python\python38\lib\site-packages (from protobuf) (49.2.1)  
Requirement already satisfied: six>=1.9 in z:\usr\lib\python3\dist-packages (fro
m protobuf) (1.14.0)
WARNING: You are using pip version 20.2.3; however, version 21.0.1 is available.
You should consider upgrading via the 'c:\users\root\local settings\application 
data\Programs\Python\Python38\python.exe -m pip install --upgrade pip' command.
root@ae099e3243a6:/mcsema/local#
root@ae099e3243a6:/mcsema/local#
root@ae099e3243a6:/mcsema/local# DISPLAY=:0.0 WINEPREFIX=~/.wine64 wine '/mcsema/ida/IDA Pro 7.5/idat64.exe' -S"/opt/trailofbits/lib/python3/site-packages/mcsema_disass-3.1.3.8-py3.8.egg/mcsema_disass/ida7/get_cfg.py --output test3.cfg --log_file log --arch amd64 --os linux --entrypoint main" test3
Error given in TUI mode: AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key
root@ae099e3243a6:/mcsema/local#
root@ae099e3243a6:/mcsema/local#
root@ae099e3243a6:/mcsema/local# DISPLAY=:0.0 WINEPREFIX=~/.wine64 wine pip install --upgrade protobuf
Collecting protobuf
  Downloading protobuf-3.15.5-py2.py3-none-any.whl (173 kB)
     |????????????????????????????????| 173 kB 3.1 MB/s
Requirement already satisfied, skipping upgrade: six>=1.9 in z:\usr\lib\python3\
dist-packages (from protobuf) (1.14.0)
Installing collected packages: protobuf
  Attempting uninstall: protobuf
    Found existing installation: protobuf 3.2.0
    Uninstalling protobuf-3.2.0:
      Successfully uninstalled protobuf-3.2.0

mcsema-disass 3.1.3.8 requires protobuf==3.2.0, but you'll have protobuf 3.15.5 
which is incompatible.
Successfully installed protobuf-3.15.5
root@ae099e3243a6:/mcsema/local#
root@ae099e3243a6:/mcsema/local#

Even giving this error, the IDA Pro worked (because the error had been fixed by upgrading protobuf) and mcsema-lift also worked perfectly.

@pgoodman When running the lifted binaries with KLEE, it just runs forever and it uses all my RAM (16GB) :/ In 15 seconds my RAM and SWAP is all filled up. I have done all the steps with LLVM-9 tools, including clang, mcsema docker image, and KLEE docker image with LLVM 9 (version 2.2). I have even downloaded the CFG of the example (https://github.com/lifting-bits/mcsema/tree/master/examples/Maze) and it runs forever too. Forever, in the sense that I have to cancel the execution before all my RAM and SWAP are used or I have to force reboot my computer.

[ekilmer]

Even giving this error, the IDA Pro worked (because the error had been fixed by upgrading protobuf) and mcsema-lift also worked perfectly.

🎉 We should remove the exact version pin for the Python tool and either set no version constraints or at least constrain it to version 3.

[FredyR4zox]

@pgoodman Any news on the execution of KLEE?

[Brubbish]

@FredyR4zox how do you run mcsema_disass with IDA Pro in wine? I tried the command like this :

DISPLAY=:0.0 WINEPREFIX=/home/bruce/.wine wine '/home/bruce/.wine/drive_c/IDA7.2/idat64.exe' -S"/home/bruce/Desktop/dir/mcsema/tools/mcsema_disass/ida7/get_cfg.py --output xz.cfg --log_file log --arch amd64 --os linux --rebase 535822336 --entrypoint main" /tmp/xz

while the wine&idat64.exe is running, it still cannot generate .cfg (it feels like they are looping), I don't know if it has anything to do with my IDA7.2 and python2.7

[FredyR4zox]

@Brubbish Can you try and change WINEPREFIX=/home/bruce/.wine to WINEPREFIX=/home/bruce/.wine64 ?
And also try to remove the --rebase argument

[Brubbish]

@Brubbish Can you try and change WINEPREFIX=/home/bruce/.wine to WINEPREFIX=/home/bruce/.wine64 ?
And also try to remove the --rebase argument

thanks, it somehow really works.