Multiple test failures in MantaClientIT

[indianwhocodes]

Description

The following four tests in MantaClientIT have been infrequently failing while running the entire integration-test suite after #507 was merged, the tests have been sorted on the basis of the frequency of occurrence along with their respective stack traces:

Detection

It is important to note that the aforementioned failures were observed under different circumstances, the first one has been observed while executing the Jenkins job java-manta-it or while running mvn verify against the java-manta root directory while the latter 3 have been observed mostly under the Jenkins test-environment .

Test-Cases Along with Stack Trace

- MantaClientIT#canCreateStreamInOneThreadAndCloseInAnother

 Exception com.joyent.manta.exception.MantaClientEncryptionException

Message: There was a problem loading private key Exception Context: [1:mantaSdkVersion=3.4.1-SNAPSHOT] [2:key_details=key=AES, algorithm=RAW] ---------------------------------
Stacktrace:
at com.joyent.manta.client.crypto.EncryptionContext.initializeCipher(EncryptionContext.java:149)
at com.joyent.manta.client.crypto.EncryptionContext.<init>(EncryptionContext.java:102)
at com.joyent.manta.client.crypto.EncryptionContext.<init>(EncryptionContext.java:60)
at com.joyent.manta.client.crypto.EncryptingEntity.<init>(EncryptingEntity.java:104)
at com.joyent.manta.http.EncryptionHttpHelper.httpPut(EncryptionHttpHelper.java:226)
at com.joyent.manta.client.MantaClient.put(MantaClient.java:1244)
at com.joyent.manta.client.MantaClient.put(MantaClient.java:1174)
at com.joyent.manta.client.MantaClient.put(MantaClient.java:1257)
at com.joyent.manta.client.MantaClientIT.canCreateStreamInOneThreadAndCloseInAnother(MantaClientIT.java:141)

- MantaClientIT#canCopyStreamToFileAndCloseWithoutErrors

Exception java.security.ProviderException

Stacktrace:
at sun.security.pkcs11.P11Cipher.implUpdate(P11Cipher.java:614)
at sun.security.pkcs11.P11Cipher.engineUpdate(P11Cipher.java:511)
at sun.security.pkcs11.P11Cipher.engineUpdate(P11Cipher.java:499)
at javax.crypto.Cipher.update(Cipher.java:1831)
at org.bouncycastle.jcajce.io.CipherInputStream.nextChunk(Unknown Source)
at org.bouncycastle.jcajce.io.CipherInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at com.joyent.manta.client.crypto.MantaEncryptedObjectInputStream.read(MantaEncryptedObjectInputStream.java:465)
at com.joyent.manta.client.crypto.MantaEncryptedObjectInputStream.read(MantaEncryptedObjectInputStream.java:415)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2314)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
at com.joyent.manta.client.MantaClientIT.canCopyStreamToFileAndCloseWithoutErrors(MantaClientIT.java:130)

- MantaClientIT#canGetZeroByteFileAsString

Exception com.joyent.manta.exception.MantaClientEncryptionException

Message: There was a problem loading private key Exception Context: [1:mantaSdkVersion=3.4.1-SNAPSHOT] [2:key_details=key=AES, algorithm=RAW] ---------------------------------

Stacktrace:
at com.joyent.manta.client.crypto.EncryptionContext.initializeCipher(EncryptionContext.java:149)
at com.joyent.manta.client.crypto.EncryptionContext.<init>(EncryptionContext.java:102)
at com.joyent.manta.client.crypto.EncryptionContext.<init>(EncryptionContext.java:60)
at com.joyent.manta.client.crypto.EncryptingEntity.<init>(EncryptingEntity.java:104)
at com.joyent.manta.http.EncryptionHttpHelper.httpPut(EncryptionHttpHelper.java:226)
at com.joyent.manta.client.MantaClient.put(MantaClient.java:1624)
at com.joyent.manta.client.MantaClient.put(MantaClient.java:1569)
at com.joyent.manta.client.MantaClientIT.canGetZeroByteFileAsString(MantaClientIT.java:249) 

- MantaClientIT#canGetZeroByteFileAsInputStream

Exception com.joyent.manta.exception.MantaClientEncryptionException

Message: There was a problem loading private key Exception Context: [1:mantaSdkVersion=3.4.1-SNAPSHOT] [2:key_details=key=AES, algorithm=RAW] ---------------------------------

Stacktrace:
at com.joyent.manta.client.crypto.EncryptionContext.initializeCipher(EncryptionContext.java:149)
at com.joyent.manta.client.crypto.EncryptionContext.<init>(EncryptionContext.java:102)
at com.joyent.manta.client.crypto.EncryptionContext.<init>(EncryptionContext.java:60)
at com.joyent.manta.client.crypto.EncryptingEntity.<init>(EncryptingEntity.java:104)
at com.joyent.manta.http.EncryptionHttpHelper.httpPut(EncryptionHttpHelper.java:226)
at com.joyent.manta.client.MantaClient.put(MantaClient.java:1624)
at com.joyent.manta.client.MantaClient.put(MantaClient.java:1569)
at com.joyent.manta.client.MantaClientIT.canGetZeroByteFileAsInputStream(MantaClientIT.java:234)

[indianwhocodes]

It seems the problem we are seeing in #545 traces back to the problem we had in #257. To provide some background:

It seems we encounter partial reads of HMAC bytes frequently occurring in a specific test case MantaClientIT#canCopyStreamToFileAndCloseWithoutErrors here which subsequently raises a java.security.ProviderException with SunPKCS11-NSS and encryptionCipher being supplied here for the failing tests is AES128/CBC/PKCS5Padding i.e libnss as our security provider what's exciting is that while i was reviewing the code-history, #251 incorporated changes that comprehensively solved the problem from our end which helped us in resolving #250, which is observable in the fact that similar failures have not been observed in the other security providers or ciphers which leads us to a probable conclusion that it is a bug to be filed with libnss or the JDK

Pasted below is the exception thrown:
java.security.ProviderException: doFinal() failed
at sun.security.pkcs11.P11Cipher.implDoFinal(P11Cipher.java:776)
at sun.security.pkcs11.P11Cipher.engineDoFinal(P11Cipher.java:543)
at sun.security.pkcs11.P11Cipher.engineDoFinal(P11Cipher.java:526)
at javax.crypto.Cipher.doFinal(Cipher.java:2048)
at org.bouncycastle.jcajce.io.CipherInputStream.finaliseCipher(Unknown Source)
at org.bouncycastle.jcajce.io.CipherInputStream.close(Unknown Source)
at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:339)
at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:270)
at com.joyent.manta.client.crypto.MantaEncryptedObjectInputStream.close(MantaEncryptedObjectInputStream.java:711)
... test case frames omitted
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_OPERATION_ACTIVE
at sun.security.pkcs11.wrapper.PKCS11.C_DecryptInit(Native Method)
at sun.security.pkcs11.P11Cipher.initialize(P11Cipher.java:445)
at sun.security.pkcs11.P11Cipher.ensureInitialized(P11Cipher.java:429)
at sun.security.pkcs11.P11Cipher.implDoFinal(P11Cipher.java:744)
... common frames omitted