Outputs while running hadoop under ubuntu 16.04

Question

I am having problems running hadoop jobs in both the pseudo-cluster and cluster mode in ubuntu 16.04.

When starting the installation of vanila hadoop / hdfs - my hadoop user gets logged out and all the processes performed by this user are closed. I don't see anything in the logs (/ var / log / systemd, journalctl or dmesg) which explains why the user is logging out.

It seems like I'm not the only one who has problems with this or a similar problem:

https://stackoverflow.com/questions/38288162/in-ubuntu-16-04-running-hadoop-jar-laptop-gets-rebooted

Note: creating a special user hadoop did not actually solve the problem in my case - but limited the output for the selected user.

https://askubuntu.com/questions/784591/ubuntu-16-04-kills-session-when-resource-usage-is-extremely-high

Is it possible that some problems around the UserGroupInformation class (which may cause a logout in some circumstances), maybe some changes to systemd in ubuntu 16.04 can cause this behavior?

The last lines of the hadoop log that I get before logging out:

... 16/07/13 16:45:37 DEBUG ipc.ProtobufRpcEngine: Call: getJobReport took 4ms 16/07/13 16:45:37 DEBUG security.UserGroupInformation: PrivilegedAction as:hduser (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320) 16/07/13 16:45:37 DEBUG ipc.Client: IPC Client (1360814716) connection to laptop/127.0.1.1:37339 from hduser sending #375 16/07/13 16:45:37 DEBUG ipc.Client: IPC Client (1360814716) connection to laptop/127.0.1.1:37339 from hduser got value #375 16/07/13 16:45:37 DEBUG ipc.ProtobufRpcEngine: Call: getJobReport took 2ms Terminated hduser@laptop :~$ 16/07/13 16:45:37 DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@4e7ab839 exit 

journalctl:

 Jul 12 16:06:44 laptop systemd-logind[978]: Removed session 7. Jul 12 16:06:44 laptop systemd-logind[978]: Removed session 6. Jul 12 16:06:44 laptop systemd-logind[978]: Removed session 5. Jul 12 16:06:44 laptop systemd-logind[978]: Removed session 8. 

Syslog:

 Jul 12 16:06:43 laptop systemd[4172]: Stopped target Default. Jul 12 16:06:43 laptop systemd[4172]: Reached target Shutdown. Jul 12 16:06:44 laptop systemd[4172]: Starting Exit the Session... Jul 12 16:06:44 laptop systemd[4172]: Stopped target Basic System. Jul 12 16:06:44 laptop systemd[4172]: Stopped target Sockets. Jul 12 16:06:44 laptop systemd[4172]: Stopped target Paths. Jul 12 16:06:44 laptop systemd[4172]: Stopped target Timers. Jul 12 16:06:44 laptop systemd[4172]: Received SIGRTMIN+24 from PID 10101 (kill). Jul 12 16:06:44 laptop systemd[1]: Stopped User Manager for UID 1001. Jul 12 16:06:44 laptop systemd[1]: Removed slice User Slice of hduser. 

Answer 1

I also had a problem. It took me a while, but I found a solution here: https://unix.stackexchange.com/questions/293069/all-services-of-a-user-are-killed-when-running-multiple-services-under-this- user

In principle, some operations with androids simply stop, because why not. But systemd seems to kill the entire user process when it sees that the service process is dying.

The fix is ​​to add

 [login] KillUserProcesses=no 

to /etc/systemd/logind.conf and reboot.

I had a multiple version of ubuntu to debug the problem, and the fix seems to work only on ubuntu 16.04.

Answer 2

I had the same problem. Finally, I found that / bin / kill in ubuntu16.04 has an error in the kill process group that can solve this problem.

If pid is less than -1, then sig is sent to each process in the process group whose identifier is -pid

Due to an error in procps-ng-3.3.10, kill a process group whose identifier starts with 1, bin/yarn application -kill AppID , and the user will exit.

The problem is solved after replacing /bin/kill with a new kill compiled from procps-ng-3.3.12.

 tar xJf procps-ng-3.3.12.tar.xz cd procps-ng-3.3.12 ./configure sudo cp .lib/kill /bin/kill sudo chown root:root /bin/kill sudo cp proc/.libs/libprocps.so.6.0.0 /lib/x86_64-linux/gnu/ sudo chown root:root /lib/x86_64-linux-gnu/libprocps.so.6.0.0 

Answer 3

I had the same problem. I used Apache APEX, which is native. When killing any APEX application, my system used to log in.

Solution: Replace the kill file (present in / bin / kill) of Ubuntu 16 with the kill file of Ubuntu 14.

Everything works smoothly, as before updating for me.