01 July 2013

469. Intel compiler (icc, icpc, ifort) on Debian

I've heard it said that MKL is faster than ACML even on AMD cpus. I've also heard it said that Intel compile + mkl beats everything else, even on AMD cpus.

So let's test the veracity of that statement. I'm in particular looking forward to seeing how this affects my amd fx 8150.

Note that the ACML libs are available as separate packages for different compilers -- download the right libs when linking (i.e. the gnu ones for gcc, and the intel ones for intel composer)

But first we need to install and set-up the intel compiler suite, and that's what we'll do in this post.

In the example below I've installed in on an AMD Athlon II X3, hence the message about non-Intel architecture.

Installation:

Register for the Intel Parallel Studio XE as touched upon in this post: http://verahill.blogspot.com.au/2013/06/465-intel-mkl-math-kernel-library-on.html

Download. It's about 2 Gb. Then extract, and run install.sh

sudo apt-get install build-essential
sudo sh install.sh
Step no: 1 of 7 | Welcome -------------------------------------------------------------------------------- Welcome to the Intel(R) Parallel Studio XE 2013 Update 3 for Linux* installation program. -------------------------------------------------------------------------------- You will complete the steps below during this installation: Step 1 : Welcome Step 2 : License Step 3 : Activation Step 4 : Intel(R) Software Improvement Program Step 5 : Options Step 6 : Installation Step 7 : Complete -------------------------------------------------------------------------------- Press "Enter" key to continue or "q" to quit: -------------------------------------------------------------------------------- Checking the prerequisites. It can take several minutes. Please wait... -------------------------------------------------------------------------------- Step no: 1 of 7 | Options > Missing Optional Pre-requisite(s) -------------------------------------------------------------------------------- There are one or more optional unresolved issues. It is highly recommended to resolve them all before you continue the installation. You can fix them without exiting from the installation and re-check. Or you can quit from the installation, fix them and run the installation again. -------------------------------------------------------------------------------- Missing optional pre-requisites -- Intel(R) VTune(TM) Amplifier XE 2013 Update 5: unsupported OS -- Intel(R) Inspector XE 2013 Update 5: unsupported OS -- Intel(R) Advisor XE 2013 Update 2: unsupported OS -- Intel(R) Composer XE 2013 Update 3 for Linux*: unsupported OS -------------------------------------------------------------------------------- 1. Skip missing optional pre-requisites [default] 2. Show the detailed info about issue(s) 3. Re-check the pre-requisites h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: Step no: 2 of 7 | License -------------------------------------------------------------------------------- As noted in the Intel(R) Software Development Product End User License Agreement, the Intel(R) Software Development Product you install will send Intel [..] -------------------------------------------------------------------------------- Do you agree to be bound by the terms and conditions of this license agreement? Type "accept" to continue or "decline" to back to the previous menu: accept Step no: 3 of 7 | Activation -------------------------------------------------------------------------------- If you have purchased this product and have the serial number and a connection to the internet you can choose to activate the product at this time. Activation is a secure and anonymous one-time process that verifies your software licensing rights to use the product. Alternatively, you can choose to evaluate the product or defer activation by choosing the evaluate option. Evaluation software will time out in about one month. Also you can use license file, license manager, or remote activation if the system you are installing on does not have internet access activation options. -------------------------------------------------------------------------------- 1. I want to activate my product using a serial number [default] 2. I want to evaluate my product or activate later 3. I want to activate either remotely, or by using a license file, or by using a license manager h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: Note: Press "Enter" key to back to the previous menu. Please type your serial number (the format is XXXX-XXXXXXXX): -------------------------------------------------------------------------------- Activation completed successfully. -------------------------------------------------------------------------------- Press "Enter" key to continue: Step no: 4 of 7 | Intel(R) Software Improvement Program -------------------------------------------------------------------------------- Help improve your experience with Intel(R) software Participate in the design of future Intel software. Select 'Yes' to give us permission to learn about how you use your Intel software and we will do the rest. - No Personal contact information is collected - There are no surveys or additional follow-up emails by opting in - You can stop participating at any time Learn more about Intel(R) Software Improvement Program http://software.intel.com/en-us/articles/software-improvement-program With your permission, Intel may automatically receive anonymous information about how you use your current and future Intel software. -------------------------------------------------------------------------------- 1. Yes, I am willing to participate and improve Intel software. (Recommended) 2. No, I don't want to participate in the Intel(R) Software Improvement Program at this time. b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection: Step no: 5 of 7 | Options -------------------------------------------------------------------------------- You are now ready to begin installation. You can use all default installation settings by simply choosing the "Start installation Now" option or you can customize these settings by selecting any of the change options given below first. You can view a summary of the settings by selecting "Show pre-install summary". -------------------------------------------------------------------------------- 1. Start installation Now 2. Change install directory [ /opt/intel ] 3. Change components to install [ All ] 4. Change advanced options 5. Show pre-install summary h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: -------------------------------------------------------------------------------- Checking the prerequisites. It can take several minutes. Please wait... -------------------------------------------------------------------------------- Step no: 5 of 7 | Options > Missing Optional Pre-requisite(s) -------------------------------------------------------------------------------- There are one or more optional unresolved issues. It is highly recommended to resolve them all before you continue the installation. You can fix them without exiting from the installation and re-check. Or you can quit from the installation, fix them and run the installation again. -------------------------------------------------------------------------------- Missing optional pre-requisites -- Intel(R) VTune(TM) Amplifier XE 2013 Update 5: The system does not have an Intel Architecture processor -------------------------------------------------------------------------------- 1. Skip missing optional pre-requisites [default] 2. Show the detailed info about issue(s) 3. Re-check the pre-requisites h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: Step no: 7 of 7 | Complete -------------------------------------------------------------------------------- Thank you for installing and using the Intel(R) Parallel Studio XE 2013 Update 3 for Linux* Reminder: Intel(R) VTune(TM) Amplifier XE users must be members of the "vtune" permissions group in order to use Event-based Sampling. To register your product purchase, visit https://registrationcenter.intel.com/RegCenter/registerexpress.aspx?clientsn=N43 3-4XGWTJLB To get started using Intel(R) VTune(TM) Amplifier XE 2013 Update 5: - To set your environment variables: source /opt/intel/vtune_amplifier_xe_2013/amplxe-vars.sh - To start the graphical user interface: amplxe-gui - To use the command-line interface: amplxe-cl - For more getting started resources: /opt/intel/vtune_amplifier_xe_2013/ documentation/en/welcomepage/get_started.html. To get started using Intel(R) Inspector XE 2013 Update 5: - To set your environment variables: source /opt/intel/inspector_xe_2013/inspxe-vars.sh - To start the graphical user interface: inspxe-gui - To use the command-line interface: inspxe-cl - For more getting started resources: /opt/intel/inspector_xe_2013/ documentation/en/welcomepage/get_started.html. To get started using Intel(R) Advisor XE 2013 Update 2: - To set your environment variables: source /opt/intel/advisor_xe_2013/advixe-vars.sh - To start the graphical user interface: advixe-gui - To use the command-line interface: advixe-cl - For more getting started resources: /opt/intel/advisor_xe_2013/ documentation/en/welcomepage/get_started.html. To get started using Intel(R) Composer XE 2013 Update 3 for Linux*: - Set the environment variables for a terminal window using one of the following (replace "intel64" with "ia32" if you are using a 32-bit platform). For csh/tcsh: $ source /opt/intel/bin/compilervars.csh intel64 For bash: $ source /opt/intel/bin/compilervars.sh intel64 To invoke the installed compilers: For C++: icpc For C: icc For Fortran: ifort To get help, append the -help option or precede with the man command. - For more getting started resources: /opt/intel/composer_xe_2013/Documentation/en_US/get_started_lc.htm. /opt/intel/composer_xe_2013/Documentation/en_US/get_started_lf.htm. To view movies and additional training, visit http://www.intel.com/software/products. -------------------------------------------------------------------------------- q. Quit [default] -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [q]:


The Files and Setup:
The compiler binaries are now found in /opt/intel/composer_xe_2013.3.163/bin/intel64/ . Of particular interest are ifort, icc and icpc (fortran, c and c++).

In addition, you'll need the lib and include files, which are found in /opt/intel/composer_xe_2013.3.163/compiler/lib/intel64/ and /opt/intel/composer_xe_2013.3.163/compiler/include/intel64.

You can either simply add the libs using LD_LIBRARY_PATH, but a perhaps easier and better method is to create a file: /etc/ld.so.conf.d/intel.conf
/opt/intel/composer_xe_2013.3.163/compiler/lib/intel64 /opt/intel/composer_xe_2013.4.183/mkl/lib/intel64
Once that's done, run
sudo ldconfig

Then do
echo 'PATH=$PATH:/opt/intel/composer_xe_2013.3.163/bin/intel64' >> ~/.bashrc
source ~/.bashrc

Testing:
See this post for an example of how to compile nwchem using ifort: http://verahill.blogspot.com.au/2013/07/470-very-briefly-compiling-nwchem-63.html

468. Kernel 3.10 on Debian

NOTE I: As of 3.10.2 the nvidia module will still not build.  I've also tried 3.10.5 and it also does not work.

NOTE II: I'm getting random slowdowns on my SL410 laptop with intel graphics. Not sure if it's the same issue as this: http://verahill.blogspot.com.au/2013/03/368-slow-mouse-and-keyboard-triggered.html
Once kworker shows up in top everything grinds to a slow crawl. I also notice that I never used 3.9 on that laptop, so the issue may be present there too.

There are several ways of building a kernel. The easiest (a purely subjective statement) is to use kernel-package i.e. make-kpkg. However, every now and again I see people writing that it's been deprecated.

Either way, start by doing
sudo apt-get install fakeroot build-essential ncurses-dev
mkdir ~/tmp
cd ~/tmp
wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.tar.xz
tar xvf linux-3.10.tar.xz
cd linux-3.10/
cat /boot/config-`uname -r`>.config
make oldconfig
Timer tick handling 1. Periodic timer ticks (constant rate, no dynticks) (HZ_PERIODIC) (NEW) > 2. Idle dynticks system (tickless idle) (NO_HZ_IDLE) (NEW) 3. Full dynticks system (tickless) (NO_HZ_FULL) (NEW) choice[1-3]: Memory placement aware NUMA scheduler (NUMA_BALANCING) [N/y/?] (NEW) Simple CPU accounting cgroup subsystem (CGROUP_CPUACCT) [N/y/?] (NEW) Group CPU scheduler (CGROUP_SCHED) [N/y/?] (NEW) Automatic process group scheduling (SCHED_AUTOGROUP) [N/y/?] (NEW) Choose SLAB allocator 1. SLAB (SLAB) (NEW) > 2. SLUB (Unqueued Allocator) (SLUB) choice[1-2?]: Linux guest support (HYPERVISOR_GUEST) [N/y/?] (NEW) Timer frequency 1. 100 HZ (HZ_100) 2. 250 HZ (HZ_250) (NEW) 3. 300 HZ (HZ_300) > 4. 1000 HZ (HZ_1000) choice[1-4?]: Memory Hotplug (ACPI_HOTPLUG_MEMORY) [N/y/?] (NEW) AMD frequency sensitivity feedback powersave bias (X86_AMD_FREQ_SENSITIVITY) [N/m/?] (NEW) Kernel support for scripts starting with #! (BINFMT_SCRIPT) [Y/n/m/?] (NEW) InfiniBand media type support (TIPC_MEDIA_IB) [N/y/?] (NEW) Network Coding (BATMAN_ADV_NC) [N/y/?] (NEW) NETLINK: mmaped IO (NETLINK_MMAP) [N/y/?] (NEW) NETLINK: socket monitoring interface (NETLINK_DIAG) [N/m/y/?] (NEW) Dummy IRQ handler (DUMMY_IRQ) [N/m/y/?] (NEW) Generic on-chip SRAM driver (SRAM) [N/y/?] (NEW) ME Enabled Intel Chipsets (INTEL_MEI_ME) [N/m/y/?] (NEW) Block device as cache (BCACHE) [N/m/y/?] (NEW) Qualcomm Atheros AR816x/AR817x support (ALX) [N/m/y/?] (NEW) QLOGIC QLCNIC 83XX family SR-IOV Support (QLCNIC_SRIOV) [Y/n/?] (NEW) Realtek RTL8152 Based USB 2.0 Ethernet Adapters (USB_RTL8152) [N/m/?] (NEW) Atheros ath9k rate control (ATH9K_LEGACY_RATE_CONTROL) [N/y/?] (NEW) rt2800usb - Include support for rt55xx devices (EXPERIMENTAL) (RT2800USB_RT55XX) [N/y/?] (NEW) y Realtek RTL8188EE Wireless Network Adapter (RTL8188EE) [N/m/?] (NEW) IMS Passenger Control Unit driver (INPUT_IMS_PCU) [N/m/?] (NEW) Qualcomm Single-wire Serial Bus Interface (SSBI) (SSBI) [N/m/y/?] (NEW) Analog Devices ADT7310/ADT7320 (SENSORS_ADT7310) [N/m/y/?] (NEW) National Semiconductor LM95234 (SENSORS_LM95234) [N/m/?] (NEW) Nuvoton NCT6775F and compatibles (SENSORS_NCT6775) [N/m/y/?] (NEW) generic cpu cooling support (CPU_THERMAL) [N/y/?] (NEW) y ChromeOS Embedded Controller (MFD_CROS_EC) [N/m/y/?] (NEW) Silicon Laboratories 4761/64/68 AM/FM radio. (MFD_SI476X_CORE) [N/m/?] (NEW) System Controller Register R/W Based on Regmap (MFD_SYSCON) [N/y/?] (NEW) TI TPS65912 Power Management chip (MFD_TPS65912) [N/y/?] (NEW) Conexant cx25821 support (VIDEO_CX25821) [N/m/?] (NEW) Cypress firmware helper routines (CYPRESS_FIRMWARE) [N/m] (NEW) QXL virtual GPU (DRM_QXL) [N/m/?] (NEW) Apple infrared receiver (HID_APPLEIR) [N/m/?] (NEW) Enable USB persist by default (USB_DEFAULT_PERSIST) [Y/n/?] (NEW) USB-Wishbone adapter interface driver (USB_SERIAL_WISHBONE) [N/m/?] (NEW) USB Physical Layer drivers (USB_PHY) [N/y/?] (NEW) PXA 27x (USB_PXA27X) [N/m/?] (NEW) MARVELL PXA2128 USB 3.0 controller (USB_MV_U3D) [N/m/?] (NEW) LED Support for TI LP5562 LED driver chip (LEDS_LP5562) [N/m/?] (NEW) LED Camera Flash/Torch Trigger (LEDS_TRIGGER_CAMERA) [N/m/y/?] (NEW) y iSCSI Extentions for RDMA (iSER) target support (INFINIBAND_ISERT) [N/m/?] (NEW) Set system time from RTC on startup and resume (RTC_HCTOSYS) [Y/n/?] (NEW) Set the RTC time based on NTP synchronization (RTC_SYSTOHC) [Y/n/?] (NEW) RTC used to set the system time (RTC_HCTOSYS_DEVICE) [rtc0] (NEW) WIS GO7007 MPEG encoder support (VIDEO_GO7007) [N/m/?] (NEW) DesignWare USB2 DRD Core Support (USB_DWC2) [N/m/?] (NEW) pvpanic device support (PVPANIC) [N/m/y/?] (NEW) Reset Controller Support (RESET_CONTROLLER) [N/y/?] (NEW) XFS Verbose Warnings (XFS_WARN) [N/y/?] (NEW) Btrfs will run sanity tests upon loading (BTRFS_FS_RUN_SANITY_TESTS) [N/y/?] (NEW) Btrfs debugging support (BTRFS_DEBUG) [N/y/?] (NEW) EFI Variable filesystem (EFIVAR_FS) [N/m/y/?] (NEW) torture tests for RCU (RCU_TORTURE_TEST) [N/m/y/?] (NEW) Ring buffer startup self test (RING_BUFFER_STARTUP_TEST) [N/y/?] (NEW) Test functions located in the string_helpers module at runtime (TEST_STRING_HELPERS) [N/m/y] (NEW) CMAC support (CRYPTO_CMAC) [N/m/y/?] (NEW) SHA256 digest algorithm (SSSE3/AVX/AVX2) (CRYPTO_SHA256_SSSE3) [N/m/y/?] (NEW) y SHA512 digest algorithm (SSSE3/AVX/AVX2) (CRYPTO_SHA512_SSSE3) [N/m/y/?] (NEW) y Camellia cipher algorithm (x86_64/AES-NI/AVX2) (CRYPTO_CAMELLIA_AESNI_AVX2_X86_64) [N/m/y/?] (NEW) m Serpent cipher algorithm (x86_64/AVX2) (CRYPTO_SERPENT_AVX2_X86_64) [N/m/y/?] (NEW) m KVM legacy PCI device assignment support (KVM_DEVICE_ASSIGNMENT) [Y/n/?] (NEW) VHOST_SCSI TCM fabric driver (VHOST_SCSI) [N/m/?] (NEW)
make menuconfig

You can now enable any additional modules by navigating the menu structure. Note that most likely you don't have to enable anything in this step, but it can come in handy if there's a major transition (e.g. the way multimedia was handled changed between kernel 3.5 and 3.6) or if you want to enable a previously disabled option.

Then pick either method 1 or 2 below.

If you only want to compile modules that are currently in use (not a good idea if you want to use the same kernel on a range of computers, or have USB devices that aren't currently plugged in) you can do that by using make localmodconfig instead of make oldconfig. I wouldn't recommend it -- in most cases it won't make a faster kernel, and space and memory tends not to be much of in the way of issues these days.


Method 1. kernel-package
Below, change -j2 to -jX, where X is the number of cores in your CPU (not cores+1 or anything funny like that. See other posts on this blog for compilation performance tests)

sudo apt-get install kernel-package
make-kpkg clean
time fakeroot make-kpkg -j2 --initrd kernel_image kernel_headers
sudo dpkg -i ../linux-image-3.10.0_3.10.0-10.00.Custom_amd64.deb ../linux-headers-3.10.0_3.10.0-10.00.Custom_amd64.deb


Took 49 minutes on a 3-core AMD Athlon II, and used ca 7 Gb.

The files are shown below:
-rw-r--r--  1 me me 8.4M Jul  1 16:02 linux-headers-3.10.0_3.10.0-10.00.Custom_amd64.deb
-rw-r--r--  1 me me  32M Jul  1 16:00 linux-image-3.10.0_3.10.0-10.00.Custom_amd64.deb


Method 2. make deb-pkg

make clean
time make deb-pkg -j2
sudo dpkg -i ../linux-firmware-image_3.10.0-2_amd64.deb ../linux-headers-3.10.0_3.10.0-2_amd64.deb ../linux-libc-dev_3.10.0-2_amd64.deb ../linux-image-3.10.0_3.10.0-2_amd64.deb

It took ca 50 minutes, it used ca 7 Gb, and generated the following files:
-rw-r--r-- 1 me me 1.1M Jul  1 16:58 ../linux-firmware-image_3.10.0-2_amd64.deb
-rw-r--r-- 1 me me 9.7M Jul  1 16:58 ../linux-headers-3.10.0_3.10.0-2_amd64.deb
-rw-r--r-- 1 me me 458M Jul  1 17:13 ../linux-image-3.10.0_3.10.0-2_amd64.deb
-rw-r--r-- 1 me me 920K Jul  1 16:58 ../linux-libc-dev_3.10.0-2_amd64.deb

Note the size of the linux-image-3.10.0_3.10.0-2_amd64.deb package.


Difference:
There are a few differences. One method, the kpgk one, is supposedly 'deprecated', but it's been working fine in the past and will work fine in the future (at least for some time).

The make dev-pkg method also generates a much bigger image file -- by a factor of 15 or so.

Otherwise, the chief difference, from what I can see, is if you want to uninstall the kernel. Remove the libc-dev package requires you to downgrade the package using apt-get, by specifying a version e.g.

apt-cache policy linux-libc-dev
linux-libc-dev: Installed: 3.10.0-1 Candidate: 3.10.0-1 Version table: *** 3.10.0-1 0 100 /var/lib/dpkg/status 3.9.6-1~bpo70+1 0 100 http://ftp.iinet.net.au/debian/debian/ wheezy-backports/main amd64 Packages 100 http://ftp.debian.org/debian/ wheezy-backports/main amd64 Packages 3.2.46-1 0 500 http://ftp.iinet.net.au/debian/debian/ wheezy/main amd64 Packages 3.2.41-2+deb7u2 0 500 http://security.debian.org/ wheezy/updates/main amd64 Packages
sudo apt-get install linux-libc-dev=3.9.6-1~bpo70+1

I personally prefer the kernel-package approach.

467 wget and tor issue

Ever since I set up Tor on my debian workstation I've been having issues using wget:
wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.tar.xz
--2013-07-01 11:32:35-- https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.tar.xz Connecting to 127.0.0.1:9050... connected. Proxy tunneling failed: Tor is not an HTTP ProxyUnable to establish SSL connection.
For fun I also tried torify although I don't want to download the kernel via Tor:

torify wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.tar.xz
--2013-07-01 11:36:20-- https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.tar.xz Connecting to 127.0.0.1:9050... 11:36:20 libtorsocks(26463): connect: Connection is to a local address (127.0.0.1), may be a TCP DNS request to a local DNS server so have to reject to be safe. Please report a bug to http://code.google.com/p/torsocks/issues/entry if this is preventing a program from working properly with torsocks. failed: No such file or directory. Retrying.
Note that I DON'T want to use wget with Tor. I don't want to eat up bandwidth on the Tor network for stuff like this. When I use wget I want to use a direct connection.

I haven't configured /etc/wgetrc and so I was a bit surprised that this kept on happening.

The solution:
edit /etc/wgetrc and put
use_proxy=off

anywhere. And you're done.